Files
home-network-k3s/docs/08-tls-and-certificates.md
T
jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:46:00 -06:00

5.4 KiB

08 — TLS & Certificates

Serves: R-TLS-01/02/03, R-DNS-01. Related: ADR-0006 (cert-manager + Let's Encrypt), ADR-0016 (Cloudflare Tunnel public ingress).

Requirement recap

Every externally reachable service presents TLS from a credible, trusted, non-self-signed authority, with certificates auto issued, renewed, and revoked — including for internal-only services (R-TLS-01/02/03).

Solution: cert-manager + Let's Encrypt via DNS-01

  • cert-manager runs in the cluster and manages the full certificate lifecycle.
  • Let's Encrypt is the trusted, non-self-signed CA (publicly trusted; free; automatable) — satisfies R-TLS-01.
  • ACME DNS-01 challenge against the keithjackson.dev DNS zone (Cloudflare or equivalent, per OQ-4). DNS-01 is the key choice because it validates domain control without exposing the service publicly — so internal-only services get real, trusted certs (R-TLS-03), and wildcard certificates (*.home.keithjackson.dev) are possible.

Issuance model

  • A ClusterIssuer for Let's Encrypt production (and one for staging, used during bring-up to avoid rate limits).
  • Certificates are requested declaratively — either a wildcard *.home.keithjackson.dev cert shared by the Gateway, or per-service certs, decided per ADR-0006. Recommendation: a wildcard cert for *.home.keithjackson.dev terminated at the Gateway simplifies onboarding (a new service needs no new cert), with per-service certs available where isolation warrants (Plex's L4 VIP name has its own).
  • The DNS provider API token is a SOPS-encrypted secret (07).

DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token)

Certificate issuance needs programmatic write access to the zone: ACME DNS-01 must write _acme-challenge TXT records, including for two-label-deep names like plex.home.keithjackson.dev.

Decision (OQ-4): use a scoped Cloudflare API token — Zone·DNS·Edit limited to the keithjackson.dev zone — stored SOPS-encrypted (07). With it:

  • cert-manager solves DNS-01 by writing/removing TXT challenge records automatically (issue, renew, revoke — R-TLS-02), including the wildcard *.home.keithjackson.dev cert and internal-only services (R-TLS-03).
  • ExternalDNS manages the *.home.keithjackson.dev records automatically (09, R-DNS-02) — no manual record editing for public services.
  • cloudflare-ddns keeps the DNS-only wan. record current for WireGuard (09).

Operator prerequisite: create the scoped token in Cloudflare and hand it over once; it is then committed SOPS-encrypted and never touched manually again. The token is least-privilege (single zone, DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt + commit.

Renewal & revocation (R-TLS-02)

  • cert-manager renews well before expiry automatically; no human action.
  • Revocation: removing/rotating a certificate is a declarative change; cert-manager handles issuance of replacements. Expiry monitoring is an alert condition (10, R-OBS-04).

Where TLS terminates

  • North-south: at the Gateway API listener (implemented by Cilium, 09), using the cert-manager-issued cert. Internal services resolve to the Gateway on the LAN.

  • Public services via the Cloudflare Tunnel (ADR-0016) are TLS twice:

    • Edge: Cloudflare terminates the client's TLS with an ACM/Total TLS certificate. It is publicly trusted and auto-renewed (R-TLS-01/02). Universal SSL is not enough, because it covers only one subdomain level.
    • Origin: cloudflared re-encrypts to the Gateway and validates the Let's Encrypt wildcard, using the service hostname as SNI. Verification is never disabled.

    Cloudflare sees plaintext between the two legs, a trade-off accepted in ADR-0016.

  • Internal service-to-service traffic relies primarily on network policy for security; mTLS between workloads is optional future hardening (could be added via Cilium/service mesh) and is not a launch requirement — record as a possible future ADR.

Trust chain for internal clients

Both certificates a client can meet are publicly trusted, so internal clients need no custom CA distribution. A name Pi-hole holds a local record for (09) resolves to the internal Gateway and serves the Let's Encrypt wildcard; any other public name resolves to Cloudflare's edge and serves its edge certificate.

Acceptance gates

  • https://<service>.home.keithjackson.dev serves a valid, non-self-signed, unexpired certificate (verified by an external TLS checker), including for internal-only services.
  • Certificates renew automatically ahead of expiry with no manual step (observed across a renewal).
  • Certificate expiry and issuance failures raise alerts.
  • Onboarding a new service requires no manual certificate step.

Resolved

  • OQ-4 — DNS-01 via a scoped Cloudflare API token (Zone·DNS·Edit, keithjackson.dev), SOPS-encrypted; drives cert-manager DNS-01, ExternalDNS and the wan. record updater.
  • Wildcard vs. per-service certs default — wildcard *.home.keithjackson.dev (ADR-0006).