correction: credential-forward + unified graph; activate SOPS (author-first)

Operator directive: don't diverge architecture to avoid asking for credentials; provision them upfront via SOPS and keep the existing (unified) graph. Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2; monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance. Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's; restore unified base graph (dev = ../base + cluster-vars + SOPS decryption, R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes. platform/kustomization.yaml now phased (single shared list): Phase-2 set = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components commented with phase markers. SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia); bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real secret values (operator encrypts). Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 22:09:20 -05:00
1 parent e1c738fa63
commit a108d5ab98
@@ -0,0 +1,74 @@
+# Credentials & SOPS — operator provisioning checklist
+
+All platform secrets are **SOPS-encrypted (age)**, committed to git, and decrypted by Flux at
+reconcile (ADR-0007, R-SEC-01). **Agents never handle real secret values** — encryption is done by
+you (the operator). This file is the end-to-end flow. Templates live in
+[`secret-templates/`](secret-templates/); the master `.sops.yaml` is at the **repo root**.
+
+Credentials are provisioned **upfront** so component rollout is gated only by *phase/readiness*,
+never by a missing secret.
+
+---
+
+## One-time SOPS setup (do this first)
+
+1. **Generate the age keypair** (private key stays on your machine + goes to the cluster; never in git):
+   ```bash
+   ./bootstrap/sops/age-keygen.sh          # prints the PUBLIC recipient (age1…) + writes the private key
+   ```
+2. **Fill the recipient** into the repo-root `.sops.yaml` — replace **both** `REPLACE-ME-AGE-PUBLIC-RECIPIENT`
+   occurrences with the `age1…` public key from step 1. (This file governs encryption for `platform/**`,
+   `apps/**`, `catalog/**`, and any `secrets/**`.)
+3. **Install the private key on the cluster** as the `sops-age` Secret so Flux can decrypt:
+   ```bash
+   ./bootstrap/sops/install-age-key.sh     # creates secret/sops-age in flux-system (see age-key-secret.md)
+   ```
+   The dev/prod overlays already point the `platform`/`apps` Flux Kustomizations at `decryption.secretRef: sops-age`.
+
+> Keep the age **private** key out of the repo (same discipline as the ansible-vault passphrase at
+> `~/.config/home-k3s/ansible-vault-pass`). If it leaks, rotate: new key → re-encrypt all `*.sops.yaml` → re-install.
+
+---
+
+## Per-secret flow (repeat for each secret below)
+
+```bash
+# 1. copy the template into its component dir under its real name
+cp bootstrap/sops/secret-templates/<template>.yaml.example platform/<component>/<name>.sops.yaml
+# 2. fill the real value(s) in stringData:
+$EDITOR platform/<component>/<name>.sops.yaml
+# 3. encrypt in place (SOPS picks up the root .sops.yaml rules automatically)
+sops -e -i platform/<component>/<name>.sops.yaml
+# 4. wire it: uncomment the pre-added `- <name>.sops.yaml` line in platform/<component>/kustomization.yaml
+# 5. verify it still builds, then commit the CIPHERTEXT
+kubectl kustomize platform/<component> >/dev/null && git add -A && git commit
+```
+
+---
+
+## Secret inventory (provision all now; wire per phase)
+
+| Secret | Namespace | Keys | Consumer | Phase | Template |
+|---|---|---|---|---|---|
+| `cloudflare-api-token` | `cert-manager` | `api-token` | cert-manager DNS-01 ClusterIssuer | **2** | `cloudflare-api-token.cert-manager.yaml.example` |
+| `cloudflare-api-token` | `external-dns` | `api-token` | ExternalDNS (`CF_API_TOKEN`) — **same token**, 2nd copy | **2** | `cloudflare-api-token.external-dns.yaml.example` |
+| `grafana-admin-credentials` | `monitoring` | `admin-user`, `admin-password` | Grafana admin | 4 | `grafana-admin-credentials.yaml.example` |
+| `alertmanager-smtp` | `monitoring` | `smtp-password` | Alertmanager email | 4 | `alertmanager-smtp.yaml.example` |
+| `keycloak-admin-credentials` | `keycloak` | `admin-user`, `admin-password` ⚠️verify chart keys; DB secret likely also needed | Keycloak IdP | 6 | `keycloak-admin-credentials.yaml.example` |
+| `oauth2-proxy-credentials` | `oauth2-proxy` | `client-id`, `client-secret`, `cookie-secret` ⚠️verify chart keys | oauth2-proxy forward-auth | 6 | `oauth2-proxy-credentials.yaml.example` |
+| `pia-credentials` | *(Phase-3 VPN ns, TBA)* | `username`, `password` | Gluetun VPN egress (protocol ≠ TCP) — **not wired yet** | 3 | `pia-credentials.yaml.example` |
+
+**Cloudflare token scope:** one API token with **Zone → DNS → Edit** + **Zone → Read** on the
+`keithjackson.dev` zone serves both cert-manager (DNS-01) and ExternalDNS. Same value in both namespace copies.
+
+---
+
+## Phase-2 blockers (needed to un-suspend the Phase-2 platform tier)
+
+- [ ] SOPS one-time setup (age key → `.sops.yaml` → `sops-age` on cluster).
+- [ ] `cloudflare-api-token` provisioned + wired in **both** `cert-manager` and `external-dns`.
+- [ ] ACME account email set in `platform/cert-manager/clusterissuers.yaml` (both issuers) — non-secret; a sensible default may already be filled, confirm it.
+- [ ] Push current `main` to the host Gitea (Flux source), then create the dev root Kustomization to reconcile `clusters/dev`.
+
+Later-phase secrets (Grafana/SMTP, Keycloak, oauth2-proxy, PIA) can be encrypted now and simply wired
+when their component is enabled in `platform/kustomization.yaml`.
@@ -0,0 +1,67 @@
+# SOPS Secret Templates
+
+This directory holds **reference templates** for the SOPS-encrypted secrets that
+platform components consume. They are **not applied and not encrypted**.
+
+## Why `.yaml.example`?
+
+Every file here uses a `.yaml.example` extension so that:
+
+- **kustomize never loads them** (they are not `.yaml` / not listed in any
+  `kustomization.yaml`), and
+- **SOPS never auto-encrypts them** (the repo's `.sops.yaml` creation rules match
+  `*.sops.yaml`, not `*.yaml.example`).
+
+They exist purely so the operator can see the exact Kubernetes Secret shape
+(name, namespace, keys) each component expects. **They contain only placeholder
+values** (`REPLACE-ME`, `<fill me>`) — never real secrets.
+
+## Operator flow (per secret)
+
+For each `<name>.yaml.example`:
+
+1. **Copy** it to the target path shown in the file's top comment, renaming to
+   `<name>.sops.yaml`. Example:
+
+   ```bash
+   cp bootstrap/sops/secret-templates/grafana-admin-credentials.yaml.example \
+      platform/monitoring/grafana-admin-credentials.sops.yaml
+   ```
+
+2. **Fill** in the real values (replace every `REPLACE-ME` / `<fill me>`).
+
+3. **Encrypt in place** with SOPS + age:
+
+   ```bash
+   sops -e -i platform/monitoring/grafana-admin-credentials.sops.yaml
+   ```
+
+4. **Wire it** into the component by adding the resource line (shown in the
+   file's top comment) under `resources:` in that component's
+   `kustomization.yaml`:
+
+   ```yaml
+   resources:
+     - grafana-admin-credentials.sops.yaml
+   ```
+
+5. **Commit** the encrypted `.sops.yaml` (ciphertext is safe to commit) and the
+   `kustomization.yaml` change. Flux decrypts it at reconcile via the cluster
+   age key.
+
+## Files in this directory
+
+| Template | Secret / namespace | Phase |
+|---|---|---|
+| `cloudflare-api-token.cert-manager.yaml.example` | `cloudflare-api-token` / `cert-manager` | 2 |
+| `cloudflare-api-token.external-dns.yaml.example` | `cloudflare-api-token` / `external-dns` (same token) | 2 |
+| `grafana-admin-credentials.yaml.example` | `grafana-admin-credentials` / `monitoring` | 4 |
+| `alertmanager-smtp.yaml.example` | `alertmanager-smtp` / `monitoring` | 4 |
+| `keycloak-admin-credentials.yaml.example` | `keycloak-admin-credentials` / `keycloak` | 6 |
+| `oauth2-proxy-credentials.yaml.example` | `oauth2-proxy-credentials` / `oauth2-proxy` | 6 |
+| `pia-credentials.yaml.example` | `pia-credentials` / TBD (Phase-3 VPN) | 3 |
+
+## Master checklist
+
+See [`bootstrap/sops/CREDENTIALS.md`](../CREDENTIALS.md) for the full,
+end-to-end credential provisioning flow and checklist.
@@ -0,0 +1,26 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      platform/monitoring/alertmanager-smtp.sops.yaml
+# (b) kustomization: add this line under `resources:` in
+#                    platform/monitoring/kustomization.yaml
+#                       - alertmanager-smtp.sops.yaml
+# (c) Encrypt:       sops -e -i platform/monitoring/alertmanager-smtp.sops.yaml
+# (d) Consumed by:   Alertmanager (kube-prometheus-stack). Only the SMTP
+#                    PASSWORD lives here. It is mounted at
+#                    /etc/alertmanager/secrets/alertmanager-smtp/smtp-password. [Phase 4]
+#
+# NOTE: The non-secret SMTP endpoint fields — smtp_smarthost, smtp_from,
+# smtp_auth_username, and the receiver `to:` addresses — are configured in
+# platform/monitoring/kube-prometheus-stack.yaml (currently REPLACE-ME
+# placeholders there). Only the password is this secret.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: alertmanager-smtp
+  namespace: monitoring
+type: Opaque
+stringData:
+  smtp-password: REPLACE-ME
@@ -0,0 +1,23 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      platform/cert-manager/cloudflare-api-token.sops.yaml
+# (b) kustomization: add this line under `resources:` in
+#                    platform/cert-manager/kustomization.yaml
+#                       - cloudflare-api-token.sops.yaml
+# (c) Encrypt:       sops -e -i platform/cert-manager/cloudflare-api-token.sops.yaml
+# (d) Consumed by:   cert-manager ClusterIssuers (DNS-01 solver). The Cloudflare
+#                    DNS-01 solver's apiTokenSecretRef points at secret
+#                    `cloudflare-api-token`, key `api-token`. [Phase 2]
+#
+# The token must be a Cloudflare API token with Zone:DNS:Edit + Zone:Read scope.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cloudflare-api-token
+  namespace: cert-manager
+type: Opaque
+stringData:
+  api-token: REPLACE-ME-CLOUDFLARE-DNS-EDIT-TOKEN
@@ -0,0 +1,27 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      platform/externaldns/cloudflare-api-token.sops.yaml
+#                   (component DIR is platform/externaldns/, but the Kubernetes
+#                    NAMESPACE is external-dns — note the hyphen.)
+# (b) kustomization: add this line under `resources:` in
+#                    platform/externaldns/kustomization.yaml
+#                       - cloudflare-api-token.sops.yaml
+# (c) Encrypt:       sops -e -i platform/externaldns/cloudflare-api-token.sops.yaml
+# (d) Consumed by:   external-dns reads env CF_API_TOKEN via secretKeyRef
+#                    name `cloudflare-api-token`, key `api-token`. [Phase 2]
+#
+# NOTE: This is the SAME Cloudflare token as the cert-manager copy — just a
+# second copy living in the external-dns namespace. ClusterIssuer secrets and
+# external-dns run in different namespaces, so the token must exist in both. A
+# single Cloudflare API token with Zone:DNS:Edit + Zone:Read scope serves both.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cloudflare-api-token
+  namespace: external-dns
+type: Opaque
+stringData:
+  api-token: REPLACE-ME-CLOUDFLARE-DNS-EDIT-TOKEN
@@ -0,0 +1,21 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      platform/monitoring/grafana-admin-credentials.sops.yaml
+# (b) kustomization: add this line under `resources:` in
+#                    platform/monitoring/kustomization.yaml
+#                       - grafana-admin-credentials.sops.yaml
+# (c) Encrypt:       sops -e -i platform/monitoring/grafana-admin-credentials.sops.yaml
+# (d) Consumed by:   kube-prometheus-stack -> grafana.admin.existingSecret
+#                    (userKey: admin-user, passwordKey: admin-password). [Phase 4]
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: grafana-admin-credentials
+  namespace: monitoring
+type: Opaque
+stringData:
+  admin-user: admin
+  admin-password: REPLACE-ME
@@ -0,0 +1,33 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      platform/keycloak/keycloak-admin-credentials.sops.yaml
+# (b) kustomization: add this line under `resources:` in
+#                    platform/keycloak/kustomization.yaml
+#                       - keycloak-admin-credentials.sops.yaml
+# (c) Encrypt:       sops -e -i platform/keycloak/keycloak-admin-credentials.sops.yaml
+# (d) Consumed by:   Keycloak Helm chart -> auth.existingSecret. [Phase 6]
+#
+# !! VERIFY KEY NAMES BEFORE ENCRYPTING !!
+# The exact stringData key names depend on the pinned Keycloak chart version.
+# Bitnami Keycloak commonly reads `admin-password`; some versions also read
+# `admin-user` (and older/other variants use `adminPassword`). Check the pinned
+# chart's values (auth.existingSecret / auth.passwordSecretKey) and adjust the
+# keys below to match before running `sops -e -i`.
+#
+# !! SEPARATE DB SECRET LIKELY NEEDED !!
+# The Keycloak helmrelease references "Admin + DB credentials". Keycloak needs a
+# PostgreSQL DB credential in addition to these admin creds. A separate DB secret
+# (or the chart's postgresql.auth.existingSecret) will likely be required at
+# Phase 6 — flag and provision it then.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: keycloak-admin-credentials
+  namespace: keycloak
+type: Opaque
+stringData:
+  admin-user: admin
+  admin-password: REPLACE-ME
@@ -0,0 +1,32 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml
+# (b) kustomization: add this line under `resources:` in
+#                    platform/oauth2-proxy/kustomization.yaml
+#                       - oauth2-proxy-credentials.sops.yaml
+# (c) Encrypt:       sops -e -i platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml
+# (d) Consumed by:   oauth2-proxy Helm chart -> existingSecret. [Phase 6]
+#
+# VALUE NOTES:
+#  - client-id / client-secret: come from the Keycloak OIDC client created for
+#    oauth2-proxy (Phase 6, after Keycloak is up).
+#  - cookie-secret: must be a 16, 24, or 32-byte value, base64-encoded.
+#    Generate with:  openssl rand -base64 32
+#
+# !! VERIFY KEY NAMES BEFORE ENCRYPTING !!
+# Confirm the exact stringData key names against the pinned oauth2-proxy chart's
+# existingSecret contract (common keys: client-id, client-secret, cookie-secret,
+# but chart versions vary). Adjust below to match before running `sops -e -i`.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: oauth2-proxy-credentials
+  namespace: oauth2-proxy
+type: Opaque
+stringData:
+  client-id: REPLACE-ME
+  client-secret: REPLACE-ME
+  cookie-secret: REPLACE-ME
@@ -0,0 +1,32 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize
+# and never auto-encrypted by SOPS). Reference scaffold only.
+#
+# (a) Copy to:      (TO BE DECIDED AT PHASE 3 — VPN component not authored yet)
+# (b) kustomization: DO NOT WIRE YET.
+# (c) Encrypt:       sops -e -i <target>.sops.yaml   (once the target path exists)
+# (d) Consumed by:   Phase-3 Gluetun VPN egress gateway. The consuming
+#                    component AND its namespace DO NOT EXIST yet — this template
+#                    is provisioned upfront. [Phase 3]
+#
+# !! NAMESPACE UNKNOWN — the metadata.namespace below is a placeholder. Set it to
+#    the VPN gateway's namespace when that component is authored at Phase 3. !!
+#
+# VALUE NOTES:
+#  - username: PIA (Private Internet Access) user, form pNNNNNNN.
+#  - password: PIA password.
+#  - Gluetun typically reads OPENVPN_USER / OPENVPN_PASSWORD (OpenVPN) or
+#    WireGuard credentials. Confirm the chosen protocol when wiring — the
+#    requirement says NOT TCP. The env var names / key names the Gluetun
+#    deployment expects may differ from username/password below; adjust to match
+#    the chosen protocol before encrypting.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pia-credentials
+  namespace: REPLACE-ME-PHASE-3-VPN-NAMESPACE
+type: Opaque
+stringData:
+  username: REPLACE-ME
+  password: REPLACE-ME