From a108d5ab98afc953a5f9d3fa1df3af5553391e54 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 16 Aug 2026 22:09:20 -0500 Subject: [PATCH] correction: credential-forward + unified graph; activate SOPS (author-first) Operator directive: don't diverge architecture to avoid asking for credentials; provision them upfront via SOPS and keep the existing (unified) graph. Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2; monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance. Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's; restore unified base graph (dev = ../base + cluster-vars + SOPS decryption, R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes. platform/kustomization.yaml now phased (single shared list): Phase-2 set = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components commented with phase markers. SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia); bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real secret values (operator encrypts). Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) --- DECISIONS-LOG.md | 40 ++++++++++ bootstrap/sops/CREDENTIALS.md | 74 +++++++++++++++++++ bootstrap/sops/secret-templates/README.md | 67 +++++++++++++++++ .../alertmanager-smtp.yaml.example | 26 +++++++ ...dflare-api-token.cert-manager.yaml.example | 23 ++++++ ...dflare-api-token.external-dns.yaml.example | 27 +++++++ .../grafana-admin-credentials.yaml.example | 21 ++++++ .../keycloak-admin-credentials.yaml.example | 33 +++++++++ .../oauth2-proxy-credentials.yaml.example | 32 ++++++++ .../pia-credentials.yaml.example | 32 ++++++++ clusters/base/cluster-vars.yaml | 2 - clusters/dev/kustomization.yaml | 38 +++++----- clusters/dev/rollout/cilium.yaml | 25 ------- clusters/dev/rollout/kustomization.yaml | 20 ----- clusters/dev/rollout/kyverno.yaml | 25 ------- clusters/dev/rollout/longhorn.yaml | 25 ------- clusters/dev/rollout/monitoring.yaml | 26 ------- platform/cert-manager/clusterissuers.yaml | 4 +- platform/cert-manager/kustomization.yaml | 4 + platform/externaldns/kustomization.yaml | 4 + platform/kustomization.yaml | 40 +++++----- .../monitoring/kube-prometheus-stack.yaml | 56 +++++++------- platform/monitoring/loki.yaml | 2 +- 23 files changed, 448 insertions(+), 198 deletions(-) create mode 100644 bootstrap/sops/CREDENTIALS.md create mode 100644 bootstrap/sops/secret-templates/README.md create mode 100644 bootstrap/sops/secret-templates/alertmanager-smtp.yaml.example create mode 100644 bootstrap/sops/secret-templates/cloudflare-api-token.cert-manager.yaml.example create mode 100644 bootstrap/sops/secret-templates/cloudflare-api-token.external-dns.yaml.example create mode 100644 bootstrap/sops/secret-templates/grafana-admin-credentials.yaml.example create mode 100644 bootstrap/sops/secret-templates/keycloak-admin-credentials.yaml.example create mode 100644 bootstrap/sops/secret-templates/oauth2-proxy-credentials.yaml.example create mode 100644 bootstrap/sops/secret-templates/pia-credentials.yaml.example delete mode 100644 clusters/dev/rollout/cilium.yaml delete mode 100644 clusters/dev/rollout/kustomization.yaml delete mode 100644 clusters/dev/rollout/kyverno.yaml delete mode 100644 clusters/dev/rollout/longhorn.yaml delete mode 100644 clusters/dev/rollout/monitoring.yaml diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index eab1176..84e8c10 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -152,6 +152,46 @@ until their hand-offs land. `helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization. 4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here. +## CORRECTION — credential-forward, unified graph (operator directive, 2026-08-16) + +**Operator pushback:** "Are we making implementation choices solely on the fact that you have to ask +for credentials? It is better to get the credentials upfront (encrypted file) and keep the rollout in +line with the existing architecture." **Correct — and it exposed a scope distortion.** + +- **A2 SUPERSEDED / D-P2A REVERTED.** Choosing components by "needs no operator secret" inverted the + roadmap: `roadmap/phases.md` **Phase 2 = Network, Storage, TLS, DNS, Ingress** → the credential-gated + cert-manager/externaldns/gateway ARE the Phase-2 core, while **monitoring is Phase 4** (I'd pulled it + forward). The per-component dev divergence (`clusters/dev/rollout/`) existed only to dodge asking for + the Cloudflare token. +- **Reverted to the existing architecture:** deleted `clusters/dev/rollout/` + the `$patch: delete`s; + restored the unified base graph (dev = ../base + cluster-vars patches + SOPS decryption patches, one + `platform` Kustomization, R-ENV-02 intact — **removes the earlier architect-review flag**). Reverted + the monitoring Grafana/SMTP dodges (it's Phase 4). Kept the non-credential-driven improvements: tofu + for_each/rejoin (F1/F2), and Kyverno parametrization + webhook namespace-excludes (Kyverno IS Phase 2). +- **DECISION D-P2B — phase the unified `platform/kustomization.yaml` (operator-approved).** Components + are enabled in the single shared list **per phase** (the same pattern that deferred kyverno): + Phase-2 set now active = **kyverno, cilium, longhorn, cert-manager, externaldns, gateway**; monitoring/ + image-automation (P4), velero (P4, OQ-BKP), kubevirt (P5), keycloak/oauth2-proxy (P6) are commented + with phase markers. dev+prod render the same list. Verified: `kubectl kustomize platform` renders only + the 6 Phase-2 namespaces; dev shows the unified platform+apps Kustomizations; prod builds; gate PASS. +- **DECISION D-SEC1 — provision ALL credentials upfront via SOPS; operator encrypts (both operator-chosen).** + Activated SOPS: `.sops.yaml` installed at repo root (recipient still REPLACE-ME — operator fills after + age-keygen). Authored `bootstrap/sops/secret-templates/*.yaml.example` (cloudflare-api-token ×2 ns, + grafana-admin, alertmanager-smtp, keycloak-admin, oauth2-proxy, pia — all placeholder-only, `.example` + so kustomize/SOPS ignore them) + `bootstrap/sops/CREDENTIALS.md` (end-to-end checklist). Pre-wired + **commented** `- cloudflare-api-token.sops.yaml` lines in cert-manager + externaldns kustomizations + (operator uncomments post-encrypt). Filled the non-secret ACME email (keithallenjackson@gmail.com, + confirm/change). **Agents never touch real secret values** (per `.sops.yaml`). + +### Operator TODO to reach Phase-2 deploy (see bootstrap/sops/CREDENTIALS.md) +1. age-keygen → fill root `.sops.yaml` recipient → install `sops-age` on cluster. +2. Provision `cloudflare-api-token` (Zone:DNS:Edit + Zone:Read on keithjackson.dev) in **both** + cert-manager + external-dns; uncomment the two kustomization lines. +3. Confirm ACME email. 4. Push `main` to host Gitea. 5. Create the dev root Kustomization (the un-suspend). +6. (Upfront, optional now) encrypt the Phase-3/4/6 secrets so later phases are credential-ready. +7. **Chart-key verification** (flagged in templates): keycloak + oauth2-proxy `existingSecret` key names, + Keycloak DB secret, and PIA/Gluetun protocol (≠ TCP) — verify at each component's phase. + ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed) | # | Issue | Impact | Fix | |---|---|---|---| diff --git a/bootstrap/sops/CREDENTIALS.md b/bootstrap/sops/CREDENTIALS.md new file mode 100644 index 0000000..eec4aa9 --- /dev/null +++ b/bootstrap/sops/CREDENTIALS.md @@ -0,0 +1,74 @@ +# Credentials & SOPS — operator provisioning checklist + +All platform secrets are **SOPS-encrypted (age)**, committed to git, and decrypted by Flux at +reconcile (ADR-0007, R-SEC-01). **Agents never handle real secret values** — encryption is done by +you (the operator). This file is the end-to-end flow. Templates live in +[`secret-templates/`](secret-templates/); the master `.sops.yaml` is at the **repo root**. + +Credentials are provisioned **upfront** so component rollout is gated only by *phase/readiness*, +never by a missing secret. + +--- + +## One-time SOPS setup (do this first) + +1. **Generate the age keypair** (private key stays on your machine + goes to the cluster; never in git): + ```bash + ./bootstrap/sops/age-keygen.sh # prints the PUBLIC recipient (age1…) + writes the private key + ``` +2. **Fill the recipient** into the repo-root `.sops.yaml` — replace **both** `REPLACE-ME-AGE-PUBLIC-RECIPIENT` + occurrences with the `age1…` public key from step 1. (This file governs encryption for `platform/**`, + `apps/**`, `catalog/**`, and any `secrets/**`.) +3. **Install the private key on the cluster** as the `sops-age` Secret so Flux can decrypt: + ```bash + ./bootstrap/sops/install-age-key.sh # creates secret/sops-age in flux-system (see age-key-secret.md) + ``` + The dev/prod overlays already point the `platform`/`apps` Flux Kustomizations at `decryption.secretRef: sops-age`. + +> Keep the age **private** key out of the repo (same discipline as the ansible-vault passphrase at +> `~/.config/home-k3s/ansible-vault-pass`). If it leaks, rotate: new key → re-encrypt all `*.sops.yaml` → re-install. + +--- + +## Per-secret flow (repeat for each secret below) + +```bash +# 1. copy the template into its component dir under its real name +cp bootstrap/sops/secret-templates/