home-network-k3s/docs/07-security-and-secrets.md

07 — Security & Secrets

Serves: R-SEC-01/02/03, supports all R-NET-* and R-TLS-*. Related: ADR-0007 (SOPS+age).

Security posture

Defense in depth with declarative, auditable controls:

• Network: default-deny east-west, explicit port-level allow, egress-class routing, VPN kill- switch (03). This is the primary isolation control.
• Identity/least privilege: each workload runs with a minimal ServiceAccount and only the RBAC it needs; no workload gets cluster-admin. VMs and containers alike get least network reach (R-SEC-02).
• Secrets: encrypted at rest in git; decrypted only in-cluster (R-SEC-01).
• TLS: trusted, auto-managed certificates end to end (08, R-TLS-*).
• Supply chain: pin images by digest where automation allows; prefer known-good upstreams; updates flow through policy, not ad-hoc pulls (11).
• Identity / SSO: a central IdP (Keycloak) provides single sign-on, one user directory, group-based authorization, and MFA; services authenticate via native OIDC or a per-app oauth2-proxy forward-auth gate so no management UI is unauthenticated (15, ADR-0014, ADR-0022, R-IDP-*). This is the primary authentication control, complementing network default-deny (the primary isolation control).
• Policy enforcement (admission): Kyverno runs in-cluster as an admission controller over application namespaces. It generates and backfills each namespace's default-deny baseline, rejects a malformed baseline, and rejects a workload that lacks a required egress-class/auth-mode label or uses host namespaces or privileged mode. Separate enforcing policies constrain the exact Job shapes that CI and agent credentials may create; these are security boundaries, so they fail closed and no environment overlay can downgrade them. No policy-exception mechanism is in use: controller-generated pods are mutated to carry truthful labels instead. Platform namespaces sit outside these policies and carry hand-authored equivalents. Policies are authored once and gate CI (14, ADR-0011), so invariants are enforced both before merge and at admission (defense in depth); the job-shape policies act only at admission and are unit-tested in CI. Enforcement complements — never replaces — the runtime behavior tests.
  • Plaintext-secret enforcement is split by layer (do not conflate): the "no plaintext Secret" invariant is a CI-time check (the committed Secret must carry SOPS ciphertext). It cannot be an admission rule, because Flux decrypts SOPS before the object reaches the API server — an admitted Secret never carries ciphertext, so an admission "reject plaintext" rule would reject every real Secret. Admission instead enforces SOPS provenance on application-namespace Secrets (the Secret was reconciled through the Flux/SOPS path), not ciphertext presence.
  • Host-namespace and privileged-pod control is Kyverno-based for the application tier; Pod Security Admission labels are applied selectively, on namespaces with their own need for them.
• Auditability: isolation, egress path, cert validity, and backup integrity are all demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.

Secrets management (R-SEC-01)

SOPS + age, native to Flux:

• Secrets are committed to git encrypted with age recipients; Flux decrypts them at reconciliation using an in-cluster age key. Plaintext never enters git.
• The age private key is bootstrapped into the cluster out-of-band (documented bootstrap step, one of the few sanctioned non-GitOps actions) and stored as a Kubernetes Secret Flux can read.
• Never print decrypted secret material into logs, PR text, or agent output (see AGENTS.md §5).

Secret categories to manage

• VPN credentials (PIA username/password + .ovpn; OpenVPN-UDP) — held host-side on the egress-gateway node(s) via Ansible vault (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret.
• Keycloak admin credential and per-service OIDC client secrets (15).
• DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
• Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016).
• SMTP credentials for Alertmanager email.
• Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, DB passwords).
• Per-app forward-auth gate credentials (IdP client secret + cookie secret; ADR-0022).
• WireGuard ingress server key, held in SOPS because a rebuild must restore it without re-enrolling every client (ADR-0021).
• Off-cluster backup target credentials (Cloudflare R2) for Velero and the database archives (ADR-0018).
• In-cluster AI agent credentials (ADR-0025) and infrastructure-as-code runtime secrets (state encryption, remote-state keys).
• Grafana admin, and any API tokens the C2 UI needs.

Rotation

• Rotating a secret = re-encrypting with SOPS and committing; Flux rolls it out. Where a secret change requires a workload restart, that is triggered by the reconcile (e.g. checksum annotation).
• VPN keys, DNS tokens, and SMTP creds SHOULD have a documented rotation procedure.

Alternatives considered (see ADR-0007)

• Sealed Secrets — viable; SOPS chosen for Flux-native decryption and multi-recipient/age simplicity and readable diffs.
• External Secrets Operator + external vault — heavier; deferred until a real external secret store exists. May be revisited via ADR.

RBAC & tenancy

• Single operator + platform automation; no multi-tenant RBAC now (out of scope, 00).
• Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions; application workloads get near-zero cluster permissions.
• The C2 UI's operational actions run through a constrained, audited path — not a shared admin kubeconfig handed to a browser. The Kubernetes console forwards each user's own IdP-issued token, so an action runs under that user's group-bound RBAC; the console's ServiceAccount holds no cluster rights (see known gaps).
• Human access to the Kubernetes API is Keycloak OIDC, with roles bound to IdP groups (15).

Node & host security

• Nodes baselined as code with Ansible (02); SSH key-only, minimal packages, required sysctls only.
• Host-level changes are IaC, not hand edits (drift + auditability).

Auditability requirements (R-SEC-03) — what an operator must be able to prove

• Isolation: "service X cannot reach service Y" — via Hubble flow logs + a denied-probe test.
• Egress path: "service X's public IP is the VPN's" — via an egress IP check.
• Kill-switch: "with VPN down, X leaks nothing" — via a fault-injection test.
• TLS: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert status + external check.
• Backups: "irreplaceable data is restorable" — via a DR drill.

These proofs are part of the platform acceptance suite, not one-time checks.

Acceptance gates

• No plaintext secret exists anywhere in git history (scanned).
• Flux decrypts SOPS secrets and workloads consume them without manual steps.
• Least-privilege RBAC verified (no workload has broad cluster rights).
• The five auditability proofs above can be produced on demand.