The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
5.4 KiB
08 — TLS & Certificates
Serves: R-TLS-01/02/03, R-DNS-01. Related: ADR-0006 (cert-manager + Let's Encrypt), ADR-0016 (Cloudflare Tunnel public ingress).
Requirement recap
Every externally reachable service presents TLS from a credible, trusted, non-self-signed authority, with certificates auto issued, renewed, and revoked — including for internal-only services (R-TLS-01/02/03).
Solution: cert-manager + Let's Encrypt via DNS-01
- cert-manager runs in the cluster and manages the full certificate lifecycle.
- Let's Encrypt is the trusted, non-self-signed CA (publicly trusted; free; automatable) — satisfies R-TLS-01.
- ACME DNS-01 challenge against the
keithjackson.devDNS zone (Cloudflare or equivalent, per OQ-4). DNS-01 is the key choice because it validates domain control without exposing the service publicly — so internal-only services get real, trusted certs (R-TLS-03), and wildcard certificates (*.home.keithjackson.dev) are possible.
Issuance model
- A
ClusterIssuerfor Let's Encrypt production (and one for staging, used during bring-up to avoid rate limits). - Certificates are requested declaratively — either a wildcard
*.home.keithjackson.devcert shared by the Gateway, or per-service certs, decided per ADR-0006. Recommendation: a wildcard cert for*.home.keithjackson.devterminated at the Gateway simplifies onboarding (a new service needs no new cert), with per-service certs available where isolation warrants (Plex's L4 VIP name has its own). - The DNS provider API token is a SOPS-encrypted secret (07).
DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token)
Certificate issuance needs programmatic write access to the zone: ACME DNS-01 must write
_acme-challenge TXT records, including for two-label-deep names like
plex.home.keithjackson.dev.
Decision (OQ-4): use a scoped Cloudflare API token — Zone·DNS·Edit limited to the
keithjackson.dev zone — stored SOPS-encrypted (07). With it:
- cert-manager solves DNS-01 by writing/removing TXT challenge records automatically (issue,
renew, revoke — R-TLS-02), including the wildcard
*.home.keithjackson.devcert and internal-only services (R-TLS-03). - ExternalDNS manages the
*.home.keithjackson.devrecords automatically (09, R-DNS-02) — no manual record editing for public services. - cloudflare-ddns keeps the DNS-only
wan.record current for WireGuard (09).
Operator prerequisite: create the scoped token in Cloudflare and hand it over once; it is then committed SOPS-encrypted and never touched manually again. The token is least-privilege (single zone, DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt + commit.
Renewal & revocation (R-TLS-02)
- cert-manager renews well before expiry automatically; no human action.
- Revocation: removing/rotating a certificate is a declarative change; cert-manager handles issuance of replacements. Expiry monitoring is an alert condition (10, R-OBS-04).
Where TLS terminates
-
North-south: at the Gateway API listener (implemented by Cilium, 09), using the cert-manager-issued cert. Internal services resolve to the Gateway on the LAN.
-
Public services via the Cloudflare Tunnel (ADR-0016) are TLS twice:
- Edge: Cloudflare terminates the client's TLS with an ACM/Total TLS certificate. It is publicly trusted and auto-renewed (R-TLS-01/02). Universal SSL is not enough, because it covers only one subdomain level.
- Origin: cloudflared re-encrypts to the Gateway and validates the Let's Encrypt wildcard, using the service hostname as SNI. Verification is never disabled.
Cloudflare sees plaintext between the two legs, a trade-off accepted in ADR-0016.
-
Internal service-to-service traffic relies primarily on network policy for security; mTLS between workloads is optional future hardening (could be added via Cilium/service mesh) and is not a launch requirement — record as a possible future ADR.
Trust chain for internal clients
Both certificates a client can meet are publicly trusted, so internal clients need no custom CA
distribution. A name Pi-hole holds a local record for (09) resolves to the
internal Gateway and serves the Let's Encrypt wildcard; any other public name resolves to
Cloudflare's edge and serves its edge certificate.
Acceptance gates
https://<service>.home.keithjackson.devserves a valid, non-self-signed, unexpired certificate (verified by an external TLS checker), including for internal-only services.- Certificates renew automatically ahead of expiry with no manual step (observed across a renewal).
- Certificate expiry and issuance failures raise alerts.
- Onboarding a new service requires no manual certificate step.
Resolved
- OQ-4 — DNS-01 via a scoped Cloudflare API token (
Zone·DNS·Edit,keithjackson.dev), SOPS-encrypted; drives cert-manager DNS-01, ExternalDNS and thewan.record updater. - Wildcard vs. per-service certs default — wildcard
*.home.keithjackson.dev(ADR-0006).