jakman85andClaude Opus 4.8
7598cff945
The in-cluster Gluetun-pod gateway (ADR-0013) cannot be steered: a pod's tun0 is invisible to the node datapath, and hostNetwork Gluetun hijacks the node's own default route (breaks k3s/etcd). The tunnel itself works (PIA egress validated), but not the steering. New decision (ADR-0015): move the tunnel to a HOST OpenVPN systemd service on egress-gateway node(s) (Ansible-managed, split-tunnel so it doesn't touch the node default route), steer egress-class=vpn pods with a CiliumEgressGatewayPolicy (native eBPF, no client-pod changes), and enforce the kill-switch with a host nftables rule (fail-closed independent of Cilium). PIA/OpenVPN-UDP provider decision (ADR-0013) stands; the .ovpn transfers over; PF re-solved host-side (Phase 6). Amended for consistency: ADR-0013 (client+steering superseded), decisions/README, docs/00, docs/01, docs/03, docs/07, agents/task-board. Co-Authored-By: Claude Opus 4.8 <[email protected]>
97 lines
6.0 KiB
Markdown
97 lines
6.0 KiB
Markdown
# 07 — Security & Secrets
|
|
|
|
Serves: **R-SEC-01/02/03**, supports all `R-NET-*` and `R-TLS-*`. Related:
|
|
[ADR-0007 (SOPS+age)](../decisions/adr-0007-secrets-sops-age.md).
|
|
|
|
## Security posture
|
|
|
|
Defense in depth with declarative, auditable controls:
|
|
|
|
- **Network**: default-deny east-west, explicit port-level allow, egress-class routing, VPN kill-
|
|
switch ([03](03-networking-isolation-and-egress.md)). This is the primary isolation control.
|
|
- **Identity/least privilege**: each workload runs with a minimal ServiceAccount and only the RBAC
|
|
it needs; no workload gets cluster-admin. VMs and containers alike get least network reach
|
|
(R-SEC-02).
|
|
- **Secrets**: encrypted at rest in git; decrypted only in-cluster (R-SEC-01).
|
|
- **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*).
|
|
- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
|
|
updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
|
|
- **Identity / SSO**: a central IdP (**Keycloak**) provides single sign-on, one user directory,
|
|
group-based authorization, and MFA; services authenticate via native OIDC or an oauth2-proxy
|
|
forward-auth layer so no management UI is unauthenticated ([15](15-identity-and-access.md),
|
|
[ADR-0014](../decisions/adr-0014-idp-keycloak.md), R-IDP-*). This is the primary *authentication*
|
|
control, complementing network default-deny (the primary *isolation* control).
|
|
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and
|
|
rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny
|
|
baseline, or a workload without a required egress-class/auth-mode label. The *same* policies
|
|
gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
|
so invariants are enforced both before merge and at admission (defense in depth). Enforcement
|
|
complements — never replaces — the runtime behavior tests.
|
|
- **Plaintext-secret enforcement is split by layer (do not conflate):** the "no plaintext Secret"
|
|
invariant is a **CI-time** check (the committed Secret must carry SOPS ciphertext). It **cannot**
|
|
be an admission rule, because Flux **decrypts SOPS before** the object reaches the API server —
|
|
an admitted Secret never carries ciphertext, so an admission "reject plaintext" rule would reject
|
|
*every* real Secret. Admission instead enforces **SOPS provenance** (the Secret was reconciled
|
|
through the Flux/SOPS path), not ciphertext presence.
|
|
- **Auditability**: isolation, egress path, cert validity, and backup integrity are all
|
|
demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.
|
|
|
|
## Secrets management (R-SEC-01)
|
|
|
|
**SOPS + age**, native to Flux:
|
|
- Secrets are committed to git **encrypted** with `age` recipients; Flux decrypts them at
|
|
reconciliation using an in-cluster age key. Plaintext never enters git.
|
|
- The age private key is bootstrapped into the cluster out-of-band (documented bootstrap step, one
|
|
of the few sanctioned non-GitOps actions) and stored as a Kubernetes Secret Flux can read.
|
|
- **Never** print decrypted secret material into logs, PR text, or agent output (see
|
|
[AGENTS.md §5](../AGENTS.md)).
|
|
|
|
### Secret categories to manage
|
|
- VPN credentials (PIA username/password + `.ovpn`; OpenVPN-UDP) — held **host-side on the egress-gateway
|
|
node(s) via Ansible vault** (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret.
|
|
- Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)).
|
|
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
|
- SMTP credentials for Alertmanager email.
|
|
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
|
|
- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn.
|
|
- Grafana admin, and any API tokens the C2 UI needs.
|
|
|
|
### Rotation
|
|
- Rotating a secret = re-encrypting with SOPS and committing; Flux rolls it out. Where a secret
|
|
change requires a workload restart, that is triggered by the reconcile (e.g. checksum annotation).
|
|
- VPN keys, DNS tokens, and SMTP creds SHOULD have a documented rotation procedure.
|
|
|
|
## Alternatives considered (see ADR-0007)
|
|
- **Sealed Secrets** — viable; SOPS chosen for Flux-native decryption and multi-recipient/age
|
|
simplicity and readable diffs.
|
|
- **External Secrets Operator + external vault** — heavier; deferred until a real external secret
|
|
store exists. May be revisited via ADR.
|
|
|
|
## RBAC & tenancy
|
|
- Single operator + platform automation; no multi-tenant RBAC now (out of scope, [00](00-vision-and-scope.md)).
|
|
- Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions;
|
|
application workloads get near-zero cluster permissions.
|
|
- The C2 UI's operational actions run through a constrained, audited path — not a shared admin
|
|
kubeconfig handed to a browser.
|
|
|
|
## Node & host security
|
|
- Nodes baselined via cloud-init/Ansible ([02](02-environments-and-provisioning.md)); SSH key-only,
|
|
minimal packages, required sysctls only.
|
|
- Host-level changes are IaC, not hand edits (drift + auditability).
|
|
|
|
## Auditability requirements (R-SEC-03) — what an operator must be able to prove
|
|
- **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a denied-probe test.
|
|
- **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
|
|
- **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
|
|
- **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert status +
|
|
external check.
|
|
- **Backups**: "irreplaceable data is restorable" — via a DR drill.
|
|
|
|
These proofs are part of the platform acceptance suite, not one-time checks.
|
|
|
|
## Acceptance gates
|
|
- No plaintext secret exists anywhere in git history (scanned).
|
|
- Flux decrypts SOPS secrets and workloads consume them without manual steps.
|
|
- Least-privilege RBAC verified (no workload has broad cluster rights).
|
|
- The five auditability proofs above can be produced on demand.
|