Operator directive: don't diverge architecture to avoid asking for credentials; provision them upfront via SOPS and keep the existing (unified) graph. Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2; monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance. Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's; restore unified base graph (dev = ../base + cluster-vars + SOPS decryption, R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes. platform/kustomization.yaml now phased (single shared list): Phase-2 set = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components commented with phase markers. SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia); bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real secret values (operator encrypts). Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
4.5 KiB
Credentials & SOPS — operator provisioning checklist
All platform secrets are SOPS-encrypted (age), committed to git, and decrypted by Flux at
reconcile (ADR-0007, R-SEC-01). Agents never handle real secret values — encryption is done by
you (the operator). This file is the end-to-end flow. Templates live in
secret-templates/; the master .sops.yaml is at the repo root.
Credentials are provisioned upfront so component rollout is gated only by phase/readiness, never by a missing secret.
One-time SOPS setup (do this first)
- Generate the age keypair (private key stays on your machine + goes to the cluster; never in git):
./bootstrap/sops/age-keygen.sh # prints the PUBLIC recipient (age1…) + writes the private key - Fill the recipient into the repo-root
.sops.yaml— replace bothREPLACE-ME-AGE-PUBLIC-RECIPIENToccurrences with theage1…public key from step 1. (This file governs encryption forplatform/**,apps/**,catalog/**, and anysecrets/**.) - Install the private key on the cluster as the
sops-ageSecret so Flux can decrypt:The dev/prod overlays already point the./bootstrap/sops/install-age-key.sh # creates secret/sops-age in flux-system (see age-key-secret.md)platform/appsFlux Kustomizations atdecryption.secretRef: sops-age.
Keep the age private key out of the repo (same discipline as the ansible-vault passphrase at
~/.config/home-k3s/ansible-vault-pass). If it leaks, rotate: new key → re-encrypt all*.sops.yaml→ re-install.
Per-secret flow (repeat for each secret below)
# 1. copy the template into its component dir under its real name
cp bootstrap/sops/secret-templates/<template>.yaml.example platform/<component>/<name>.sops.yaml
# 2. fill the real value(s) in stringData:
$EDITOR platform/<component>/<name>.sops.yaml
# 3. encrypt in place (SOPS picks up the root .sops.yaml rules automatically)
sops -e -i platform/<component>/<name>.sops.yaml
# 4. wire it: uncomment the pre-added `- <name>.sops.yaml` line in platform/<component>/kustomization.yaml
# 5. verify it still builds, then commit the CIPHERTEXT
kubectl kustomize platform/<component> >/dev/null && git add -A && git commit
Secret inventory (provision all now; wire per phase)
| Secret | Namespace | Keys | Consumer | Phase | Template |
|---|---|---|---|---|---|
cloudflare-api-token |
cert-manager |
api-token |
cert-manager DNS-01 ClusterIssuer | 2 | cloudflare-api-token.cert-manager.yaml.example |
cloudflare-api-token |
external-dns |
api-token |
ExternalDNS (CF_API_TOKEN) — same token, 2nd copy |
2 | cloudflare-api-token.external-dns.yaml.example |
grafana-admin-credentials |
monitoring |
admin-user, admin-password |
Grafana admin | 4 | grafana-admin-credentials.yaml.example |
alertmanager-smtp |
monitoring |
smtp-password |
Alertmanager email | 4 | alertmanager-smtp.yaml.example |
keycloak-admin-credentials |
keycloak |
admin-user, admin-password ⚠️verify chart keys; DB secret likely also needed |
Keycloak IdP | 6 | keycloak-admin-credentials.yaml.example |
oauth2-proxy-credentials |
oauth2-proxy |
client-id, client-secret, cookie-secret ⚠️verify chart keys |
oauth2-proxy forward-auth | 6 | oauth2-proxy-credentials.yaml.example |
pia-credentials |
(Phase-3 VPN ns, TBA) | username, password |
Gluetun VPN egress (protocol ≠ TCP) — not wired yet | 3 | pia-credentials.yaml.example |
Cloudflare token scope: one API token with Zone → DNS → Edit + Zone → Read on the
keithjackson.dev zone serves both cert-manager (DNS-01) and ExternalDNS. Same value in both namespace copies.
Phase-2 blockers (needed to un-suspend the Phase-2 platform tier)
- SOPS one-time setup (age key →
.sops.yaml→sops-ageon cluster). cloudflare-api-tokenprovisioned + wired in bothcert-managerandexternal-dns.- ACME account email set in
platform/cert-manager/clusterissuers.yaml(both issuers) — non-secret; a sensible default may already be filled, confirm it. - Push current
mainto the host Gitea (Flux source), then create the dev root Kustomization to reconcileclusters/dev.
Later-phase secrets (Grafana/SMTP, Keycloak, oauth2-proxy, PIA) can be encrypted now and simply wired
when their component is enabled in platform/kustomization.yaml.