diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 69da140..eab1176 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -91,6 +91,67 @@ primary play runs, so a limit that excludes the primary yields self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched. +## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16) + +Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the +no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the +running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the +GitRepository). + +### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno) +- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml` + equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env + `KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red + herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements, + cgroup, v1.16.5. **Flux adoption is a no-op reconcile.** +- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base + `platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in + the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation) + and never go healthy (`apps` dependsOn platform, wait:true). +- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in + cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready. +- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two + **by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing + mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager. +- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the + fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster. + +### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved) +Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux +Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`), +and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`. +The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it +(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created** +until their hand-offs land. +- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it). +- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges + its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim"). + This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap + patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two + `$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched** + (still builds the monolith; verified). Flag for architect ratification. + +### Component hardening (env-gated via cluster-vars so prod keeps the strict posture) +- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**), + `KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**), + `MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev + → chart-default password, dev-only/NAT-isolated). +- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/ + longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick). +- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented + with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in + UI+Grafana, just no email until the Phase-4 SMTP hand-off. +- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed), + clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid; + `make -C ci gate` PASS. + +### Still required BEFORE an actual deploy (not part of this author-first prep) +1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges). +2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes. +3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it + `helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization. +4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here. + ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed) | # | Issue | Impact | Fix | |---|---|---|---| diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index 306e58d..8558cc6 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -27,6 +27,15 @@ data: MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long MONITORING_PROMETHEUS_PVC_SIZE: "10Gi" MONITORING_LOKI_PVC_SIZE: "10Gi" + MONITORING_STORAGE_CLASS: "longhorn" # PVC tier for Prometheus/Loki (dev may override) + GRAFANA_ADMIN_SECRET: "grafana-admin-credentials" # by-name SOPS secret (prod); dev="" → chart default + + # --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture; + # the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can + # come up before policies are proven and admission is HA. --- + KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing) + KYVERNO_WEBHOOK_FAILURE_POLICY: "Fail" # dev: Ignore (fail-open during bootstrap) + KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate) # --- identity / auth (ADR-0014) --- KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA) diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index 3d47970..8312db7 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -2,39 +2,38 @@ # clusters/dev — the ACTIVE build target overlay. # # `flux bootstrap ... --path=clusters/dev` points the dev cluster's Flux here. -# This overlay adds NO resources of its own. Its ONLY job is to patch the shared cluster-vars -# ConfigMap with dev-scoped values (R-ENV-02). The base reconciliation graph is reused verbatim. +# TRANSITIONAL (Phase-2 bootstrap): this overlay reuses the base source + cluster-vars, but +# REPLACES the monolithic base `platform`/`apps` Kustomizations with a per-component NO-SECRET +# rollout (./rollout) — so dev brings up cilium/longhorn/monitoring/kyverno in order WITHOUT +# dragging in the secret-blocked components. It patches cluster-vars with dev values (R-ENV-02) +# and softens Kyverno for bootstrap. CONVERGENCE: delete ./rollout + the two $patch:delete +# below to return to the unified base graph once all operator hand-offs land. See DECISIONS-LOG. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base + - rollout patches: - # --- SOPS decryption + bootstrap source (PM-inlined from bootstrap/flux/patches/, 2026-08-16) --- - # Inlined (not cross-dir file refs) to stay within kustomize's default load restrictions. - # Realises the commented decryption TODOs in clusters/base and sets the real source URL. - # The `sops-age` Secret is created out-of-band (bootstrap/sops/install-age-key.sh) — name only. + # --- Drop the monolithic base platform/apps Kustomizations. They reconcile ALL 11 components + # as one unit (incl. secret-blocked cert-manager/externaldns/velero/keycloak/oauth2-proxy/ + # image-automation/gateway/kubevirt); ./rollout replaces them with the per-component + # no-secret set for dev bootstrap. Remove these two deletes to re-adopt the unified graph. + # (No decryption patch is needed here: the rollout components carry no SOPS content, so + # the `sops-age` Secret is not required for the dev bootstrap tier.) --- - patch: |- + $patch: delete apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: platform namespace: flux-system - spec: - decryption: - provider: sops - secretRef: - name: sops-age - patch: |- + $patch: delete apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps namespace: flux-system - spec: - decryption: - provider: sops - secretRef: - name: sops-age - patch: |- apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository @@ -84,3 +83,18 @@ patches: - op: replace path: /data/VELERO_RETENTION value: "72h" + # dev-softened Kyverno for cold-start (base defaults Enforce/Fail/3 = prod posture). + - op: replace + path: /data/KYVERNO_VALIDATION_ACTION + value: "Audit" + - op: replace + path: /data/KYVERNO_WEBHOOK_FAILURE_POLICY + value: "Ignore" + - op: replace + path: /data/KYVERNO_ADMISSION_REPLICAS + value: "1" + # dev: no Grafana admin Secret yet → empty makes the chart use its default admin password + # (dev-only, NAT-isolated). Prod keeps the by-name SOPS secret from the base default. + - op: replace + path: /data/GRAFANA_ADMIN_SECRET + value: "" diff --git a/clusters/dev/rollout/cilium.yaml b/clusters/dev/rollout/cilium.yaml new file mode 100644 index 0000000..3e4b527 --- /dev/null +++ b/clusters/dev/rollout/cilium.yaml @@ -0,0 +1,25 @@ +--- +# Cilium — ADOPT the running Ansible-installed CNI. Parity between platform/cilium and the +# live config was verified (k8sServiceHost/port via pod env, kube-proxy-replacement, gateway-api, +# egress+bpf masquerade, hubble, l2, cgroup, v1.16.5) → this reconcile is a NO-OP. +# prune: false — Cilium was bootstrapped out-of-band; adopt its config, never own-delete the CNI. +# No decryption (no SOPS content). No dependsOn (it is the datapath everything else rides on). +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: cilium + namespace: flux-system +spec: + interval: 30m + retryInterval: 2m + timeout: 10m + prune: false + wait: true + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/cilium + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/dev/rollout/kustomization.yaml b/clusters/dev/rollout/kustomization.yaml new file mode 100644 index 0000000..bc040d3 --- /dev/null +++ b/clusters/dev/rollout/kustomization.yaml @@ -0,0 +1,20 @@ +--- +# Phase-2 NO-SECRET incremental rollout — DEV bootstrap scaffold. +# +# Per-component Flux Kustomizations for the components that need NO operator secret: +# kyverno (admission, dev-softened) · cilium (adopt running CNI) · longhorn (storage) +# · monitoring (dependsOn longhorn). +# Secret-blocked components (cert-manager, externaldns, gateway, velero, keycloak, +# oauth2-proxy, image-automation, kubevirt) are intentionally NOT created until their +# operator hand-offs arrive (DECISIONS-LOG A2 / hand-off table). +# +# TRANSITIONAL: this is the dev bootstrap graph. Once every hand-off lands, dev drops +# ./rollout and re-adopts the unified base monolith (clusters/base/platform.yaml) so dev +# and prod share one reconciliation graph again (R-ENV-02 convergence). See DECISIONS-LOG. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - kyverno.yaml # first: admission present (Audit/Ignore in dev — non-blocking) before workloads + - cilium.yaml # adopt the running CNI (parity verified — no-op reconcile) + - longhorn.yaml # storage; monitoring depends on its StorageClass + - monitoring.yaml # dependsOn longhorn diff --git a/clusters/dev/rollout/kyverno.yaml b/clusters/dev/rollout/kyverno.yaml new file mode 100644 index 0000000..0e09db7 --- /dev/null +++ b/clusters/dev/rollout/kyverno.yaml @@ -0,0 +1,25 @@ +--- +# Kyverno — in-cluster admission enforcer. DEV-SOFTENED via cluster-vars: +# KYVERNO_VALIDATION_ACTION=Audit, KYVERNO_WEBHOOK_FAILURE_POLICY=Ignore, +# KYVERNO_ADMISSION_REPLICAS=1 (base defaults Enforce/Fail/3 for prod). +# Non-blocking on today's cluster (policies match only `app-namespace=true` namespaces, +# none exist yet). No SOPS content. Listed first so admission is present before workloads. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: kyverno + namespace: flux-system +spec: + interval: 1h + retryInterval: 2m + timeout: 10m + prune: true + wait: true + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/kyverno + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/dev/rollout/longhorn.yaml b/clusters/dev/rollout/longhorn.yaml new file mode 100644 index 0000000..07b7d6c --- /dev/null +++ b/clusters/dev/rollout/longhorn.yaml @@ -0,0 +1,25 @@ +--- +# Longhorn — block storage / CSI (default `longhorn` StorageClass). No SOPS content. +# No dependsOn. The bulk-NFS StorageClass in this component carries REPLACE-ME (OQ-2) but is +# an isolated, unreferenced object — it does NOT block Longhorn's HelmRelease reconcile. +# PRE-DEPLOY CHECK (runtime, out of git): confirm `iscsid` active + /var/lib/longhorn mounted +# on all 3 nodes (cloud-init provides both; verify after the T-1.5 rebuild). +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: longhorn + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 10m + prune: true + wait: true + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/longhorn + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/clusters/dev/rollout/monitoring.yaml b/clusters/dev/rollout/monitoring.yaml new file mode 100644 index 0000000..5e902f6 --- /dev/null +++ b/clusters/dev/rollout/monitoring.yaml @@ -0,0 +1,26 @@ +--- +# Monitoring — kube-prometheus-stack + Loki + Alloy. Prometheus/Loki PVCs bind the +# `longhorn` StorageClass, so this dependsOn longhorn (Flux waits for it Ready first). +# Grafana admin password (chart default in dev) and Alertmanager SMTP (deferred to the +# Phase-4 hand-off) are handled WITHOUT an operator secret in dev → NO decryption block. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: monitoring + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 15m + prune: true + wait: true + dependsOn: + - name: longhorn + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/monitoring + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars diff --git a/platform/kyverno/admission/helmrelease.yaml b/platform/kyverno/admission/helmrelease.yaml index bcce8f5..fa2b0cc 100644 --- a/platform/kyverno/admission/helmrelease.yaml +++ b/platform/kyverno/admission/helmrelease.yaml @@ -34,18 +34,32 @@ spec: remediation: retries: 3 values: - # HA admission controller so a single pod restart cannot open an enforcement gap. + # Admission controller replicas — cluster-vars knob (prod 3 / dev 1). HA so a single pod + # restart cannot open an enforcement gap; dev drops to 1 to save RAM during bootstrap. admissionController: - replicas: 3 + replicas: ${KYVERNO_ADMISSION_REPLICAS} backgroundController: enabled: true # needed for generate (default-deny baseline) + sync reportsController: enabled: true cleanupController: enabled: true - # Fail-CLOSED: if the webhook is unreachable, reject rather than silently admit - # non-conforming resources (coordination-protocol.md #8). Tune per-resource in - # the policies' webhook config if a bootstrap deadlock appears. + # Webhook failure policy — cluster-vars knob (prod Fail / dev Ignore). Fail is fail-CLOSED + # (reject rather than silently admit when the webhook is unreachable, coordination-protocol.md + # #8); dev uses Ignore (fail-open) so a cold/unhealthy Kyverno cannot brick admission during + # bootstrap. The namespaceSelector ALSO excludes system + platform namespaces at the webhook + # layer — belt-and-suspenders: policies already scope to app-namespace=true, but this + # guarantees kube-system/flux-system/CNI/storage/monitoring admission never routes to Kyverno. config: webhooks: - - failurePolicy: Fail + - failurePolicy: ${KYVERNO_WEBHOOK_FAILURE_POLICY} + namespaceSelector: + matchExpressions: + - key: kubernetes.io/metadata.name + operator: NotIn + values: + - kube-system + - flux-system + - kyverno + - longhorn-system + - monitoring diff --git a/platform/kyverno/admission/secret-provenance.yaml b/platform/kyverno/admission/secret-provenance.yaml index 988384b..64f7449 100644 --- a/platform/kyverno/admission/secret-provenance.yaml +++ b/platform/kyverno/admission/secret-provenance.yaml @@ -24,7 +24,7 @@ metadata: homelab.keithjackson.dev/requirements: R-SEC-01 homelab.keithjackson.dev/scope: admission-only spec: - validationFailureAction: Enforce + validationFailureAction: ${KYVERNO_VALIDATION_ACTION} background: false rules: - name: secrets-must-be-flux-managed diff --git a/platform/kyverno/policies/require-auth-mode-label.yaml b/platform/kyverno/policies/require-auth-mode-label.yaml index 8b182af..026dd67 100644 --- a/platform/kyverno/policies/require-auth-mode-label.yaml +++ b/platform/kyverno/policies/require-auth-mode-label.yaml @@ -25,7 +25,7 @@ metadata: Pods in application namespaces must declare homelab.keithjackson.dev/auth-mode = oidc | forward-auth | native | none. spec: - validationFailureAction: Enforce + validationFailureAction: ${KYVERNO_VALIDATION_ACTION} background: true rules: - name: require-auth-mode diff --git a/platform/kyverno/policies/require-egress-class-label.yaml b/platform/kyverno/policies/require-egress-class-label.yaml index fd58d7d..3e4d813 100644 --- a/platform/kyverno/policies/require-egress-class-label.yaml +++ b/platform/kyverno/policies/require-egress-class-label.yaml @@ -25,7 +25,7 @@ metadata: Pods in application namespaces must declare homelab.keithjackson.dev/egress-class = internet | vpn. spec: - validationFailureAction: Enforce + validationFailureAction: ${KYVERNO_VALIDATION_ACTION} background: true rules: - name: require-egress-class diff --git a/platform/kyverno/policies/restrict-host-namespaces.yaml b/platform/kyverno/policies/restrict-host-namespaces.yaml index 469d952..cf13aba 100644 --- a/platform/kyverno/policies/restrict-host-namespaces.yaml +++ b/platform/kyverno/policies/restrict-host-namespaces.yaml @@ -24,7 +24,7 @@ metadata: Application-namespace pods must not use hostNetwork/hostPID/hostIPC (which bypass default-deny) or run privileged containers. spec: - validationFailureAction: Enforce + validationFailureAction: ${KYVERNO_VALIDATION_ACTION} background: true rules: - name: disallow-host-namespaces diff --git a/platform/kyverno/policies/validate-default-deny-shape.yaml b/platform/kyverno/policies/validate-default-deny-shape.yaml index 0016d9e..8c693fa 100644 --- a/platform/kyverno/policies/validate-default-deny-shape.yaml +++ b/platform/kyverno/policies/validate-default-deny-shape.yaml @@ -20,7 +20,7 @@ metadata: to include both Ingress and Egress. Rejects a baseline that silently leaves a direction open. spec: - validationFailureAction: Enforce + validationFailureAction: ${KYVERNO_VALIDATION_ACTION} background: true rules: - name: baseline-must-deny-both-directions diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml index b12d911..14790d0 100644 --- a/platform/monitoring/kube-prometheus-stack.yaml +++ b/platform/monitoring/kube-prometheus-stack.yaml @@ -62,7 +62,7 @@ spec: storageSpec: volumeClaimTemplate: spec: - storageClassName: longhorn # replicated tier (ADR-0003) + storageClassName: ${MONITORING_STORAGE_CLASS} # replicated tier (ADR-0003); cluster-vars knob accessModes: ["ReadWriteOnce"] resources: requests: @@ -80,22 +80,24 @@ spec: memory: 32Mi limits: memory: 96Mi - # SMTP credentials are an OPERATOR HAND-OFF: a SOPS-encrypted Secret, referenced BY NAME - # only (docs/07). Mounted at /etc/alertmanager/secrets// so the config below can read - # the password from a file (never inline plaintext). - secrets: - - alertmanager-smtp - # Routing/receivers. Non-secret SMTP endpoint fields (smarthost/from/to/username) are - # operator-supplied placeholders (REPLACE-ME, same convention as velero); the PASSWORD is - # read from the mounted SOPS secret via *_file. Both severities email; subject/route differ. + # SMTP is an OPERATOR HAND-OFF DEFERRED to Phase 4 — no creds yet, and the by-name + # `alertmanager-smtp` Secret does NOT exist. Mounting a missing Secret wedges Alertmanager + # in ContainerCreating, so the mount is DISABLED until the hand-off. Re-enable verbatim + # (uncomment) when the SOPS `alertmanager-smtp` Secret is delivered. TODO(Phase-4 SMTP). + # secrets: + # - alertmanager-smtp + # Routing stays live; RECEIVERS ARE NULL until the SMTP hand-off (alerts still fire, route, + # and show in the Alertmanager UI + Grafana — they just aren't emailed). The smtp_* globals + # and email_configs are retained below, commented, so re-enabling is a pure uncomment. config: global: resolve_timeout: 5m - smtp_smarthost: "REPLACE-ME:587" # operator hand-off (e.g. smtp.gmail.com:587) - smtp_from: "REPLACE-ME" # operator hand-off (From: address) - smtp_auth_username: "REPLACE-ME" # operator hand-off (SMTP login) - smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/smtp-password - smtp_require_tls: true + # TODO(Phase-4 SMTP hand-off): uncomment together with the alertmanager-smtp Secret mount. + # smtp_smarthost: "smtp.example.com:587" + # smtp_from: "alerts@home.keithjackson.dev" + # smtp_auth_username: "" + # smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/smtp-password + # smtp_require_tls: true route: group_by: ["alertname", "namespace"] group_wait: 30s @@ -109,25 +111,29 @@ spec: group_wait: 10s repeat_interval: 1h receivers: + # NULL receivers until Phase-4 SMTP. email_configs retained (commented) for verbatim re-enable. - name: email-warning - email_configs: - - to: "REPLACE-ME" # operator hand-off (alert inbox) - send_resolved: true - headers: - subject: '[WARNING] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})' + # email_configs: + # - to: "" + # send_resolved: true + # headers: + # subject: '[WARNING] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})' - name: email-critical - email_configs: - - to: "REPLACE-ME" # operator hand-off (alert inbox) - send_resolved: true - headers: - subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})' + # email_configs: + # - to: "" + # send_resolved: true + # headers: + # subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})' # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO --- grafana: enabled: true # Admin bootstrap + (later) OIDC client are SOPS secrets referenced BY NAME (WS-SEC/ADR-0014). + # cluster-vars knob: prod = the SOPS `grafana-admin-credentials` Secret; dev = "" (empty), + # which makes the chart fall back to its default admin password (dev-only, NAT-isolated) so + # Grafana starts without an operator secret. TODO: supply a dev Secret if wider dev access. admin: - existingSecret: grafana-admin-credentials + existingSecret: ${GRAFANA_ADMIN_SECRET} userKey: admin-user passwordKey: admin-password # DEV TRIM: emptyDir (default). Dashboards are provisioned from ConfigMaps (sidecar), so no diff --git a/platform/monitoring/loki.yaml b/platform/monitoring/loki.yaml index eb44d99..5df673b 100644 --- a/platform/monitoring/loki.yaml +++ b/platform/monitoring/loki.yaml @@ -60,7 +60,7 @@ spec: replicas: 1 persistence: enabled: true - storageClass: longhorn # replicated tier (ADR-0003) + storageClass: ${MONITORING_STORAGE_CLASS} # replicated tier (ADR-0003); cluster-vars knob size: ${MONITORING_LOKI_PVC_SIZE} # overlay knob (dev 5Gi / prod large) resources: requests: