home-network-k3s/docs/07-security-and-secrets.md

# 07 — Security & Secrets

Serves: **R-SEC-01/02/03**, supports all `R-NET-*` and `R-TLS-*`. Related:
[ADR-0007 (SOPS+age)](../decisions/adr-0007-secrets-sops-age.md).

## Security posture

Defense in depth with declarative, auditable controls:

- **Network**: default-deny east-west, explicit port-level allow, egress-class routing, VPN kill-
  switch ([03](03-networking-isolation-and-egress.md)). This is the primary isolation control.
- **Identity/least privilege**: each workload runs with a minimal ServiceAccount and only the RBAC
  it needs; no workload gets cluster-admin. VMs and containers alike get least network reach
  (R-SEC-02).
- **Secrets**: encrypted at rest in git; decrypted only in-cluster (R-SEC-01).
- **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*).
- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
  updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
- **Identity / SSO**: a central IdP (**Keycloak**) provides single sign-on, one user directory,
  group-based authorization, and MFA; services authenticate via native OIDC or a per-app
  oauth2-proxy forward-auth gate so no management UI is unauthenticated
  ([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md),
  [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md), R-IDP-*). This is
  the primary *authentication* control, complementing network default-deny (the primary *isolation*
  control).
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller over
  **application namespaces**. It generates and backfills each namespace's default-deny baseline,
  rejects a malformed baseline, and rejects a workload that lacks a required egress-class/auth-mode
  label or uses host namespaces or privileged mode. Separate enforcing policies constrain the exact
  Job shapes that CI and agent credentials may create; these are security boundaries, so they fail
  closed and no environment overlay can downgrade them. No policy-exception mechanism is in use:
  controller-generated pods are mutated to carry truthful labels instead. Platform namespaces sit
  outside these policies and carry hand-authored equivalents. Policies are authored once and gate
  CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
  so invariants are enforced both before merge and at admission (defense in depth); the job-shape
  policies act only at admission and are unit-tested in CI. Enforcement complements — never
  replaces — the runtime behavior tests.
  - **Plaintext-secret enforcement is split by layer (do not conflate):** the "no plaintext Secret"
    invariant is a **CI-time** check (the committed Secret must carry SOPS ciphertext). It **cannot**
    be an admission rule, because Flux **decrypts SOPS before** the object reaches the API server —
    an admitted Secret never carries ciphertext, so an admission "reject plaintext" rule would reject
    *every* real Secret. Admission instead enforces **SOPS provenance** on application-namespace
    Secrets (the Secret was reconciled through the Flux/SOPS path), not ciphertext presence.
  - Host-namespace and privileged-pod control is Kyverno-based for the application tier; Pod
    Security Admission labels are applied selectively, on namespaces with their own need for them.
- **Auditability**: isolation, egress path, cert validity, and backup integrity are all
  demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.

## Secrets management (R-SEC-01)

**SOPS + age**, native to Flux:
- Secrets are committed to git **encrypted** with `age` recipients; Flux decrypts them at
  reconciliation using an in-cluster age key. Plaintext never enters git.
- The age private key is bootstrapped into the cluster out-of-band (documented bootstrap step, one
  of the few sanctioned non-GitOps actions) and stored as a Kubernetes Secret Flux can read.
- **Never** print decrypted secret material into logs, PR text, or agent output (see
  [AGENTS.md §5](../AGENTS.md)).

### Secret categories to manage
- VPN credentials (PIA username/password + `.ovpn`; OpenVPN-UDP) — held **host-side on the egress-gateway
  node(s) via Ansible vault** (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret.
- Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)).
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
- Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016).
- SMTP credentials for Alertmanager email.
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, DB passwords; Kasm is not onboarded).
- Per-app forward-auth gate credentials (IdP client secret + cookie secret; [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)).
- WireGuard ingress server key, held in SOPS because a rebuild must restore it without re-enrolling
  every client (ADR-0021).
- Off-cluster backup target credentials (Cloudflare R2) for Velero and the database archives (ADR-0018).
- In-cluster AI agent credentials (ADR-0025) and infrastructure-as-code runtime secrets (state
  encryption, remote-state keys).
- Grafana admin, and any API tokens the C2 UI needs.

### Rotation
- Rotating a secret = re-encrypting with SOPS and committing; Flux rolls it out. Where a secret
  change requires a workload restart, that is triggered by the reconcile (e.g. checksum annotation).
- VPN keys, DNS tokens, and SMTP creds SHOULD have a documented rotation procedure.

## Alternatives considered (see ADR-0007)
- **Sealed Secrets** — viable; SOPS chosen for Flux-native decryption and multi-recipient/age
  simplicity and readable diffs.
- **External Secrets Operator + external vault** — heavier; deferred until a real external secret
  store exists. May be revisited via ADR.

## RBAC & tenancy
- Single operator + platform automation; no multi-tenant RBAC now (out of scope, [00](00-vision-and-scope.md)).
- Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions;
  application workloads get near-zero cluster permissions.
- The C2 UI's operational actions run through a constrained, audited path — not a shared admin
  kubeconfig handed to a browser. The Kubernetes console forwards each user's own IdP-issued token,
  so an action runs under that user's group-bound RBAC; the console's ServiceAccount holds no
  cluster rights (see [known gaps](16-known-gaps.md)).
- Human access to the Kubernetes API is Keycloak OIDC, with roles bound to IdP groups
  ([15](15-identity-and-access.md)).

## Node & host security
- Nodes baselined as code with Ansible ([02](02-environments-and-provisioning.md)); SSH key-only,
  minimal packages, required sysctls only.
- Host-level changes are IaC, not hand edits (drift + auditability).

## Auditability requirements (R-SEC-03) — what an operator must be able to prove
- **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a denied-probe test.
- **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
- **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
- **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert status +
  external check.
- **Backups**: "irreplaceable data is restorable" — via a DR drill.

These proofs are part of the platform acceptance suite, not one-time checks.

## Acceptance gates
- No plaintext secret exists anywhere in git history (scanned).
- Flux decrypts SOPS secrets and workloads consume them without manual steps.
- Least-privilege RBAC verified (no workload has broad cluster rights).
- The five auditability proofs above can be produced on demand.