docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
+23
-6
@@ -8,6 +8,14 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
|
||||
> proposed ADR and escalate it, but only the **human operator/architect** approves/accepts or edits
|
||||
> ADRs. See [`../AGENTS.md` §3 and §11](../AGENTS.md).
|
||||
|
||||
**These ADRs are not the implementation's handoff log.** The dated files in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl)'s `decisions/`
|
||||
directory record what each change did and how it was verified. ADRs here record locked technology and
|
||||
pattern choices.
|
||||
|
||||
**Partial supersession.** When a new ADR replaces only part of an accepted one, the older ADR stays
|
||||
`Accepted` and its Status line names the clause superseded (the ADR-0013 precedent).
|
||||
|
||||
## Process (read before proposing a change)
|
||||
|
||||
1. To change a decision, create a **new** ADR with the next number and set `Status: Proposed`,
|
||||
@@ -42,19 +50,28 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
|
||||
|
||||
| ADR | Title | Status | Serves |
|
||||
|---|---|---|---|
|
||||
| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted | R-PLAT-01 |
|
||||
| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted — dev provisioning clause superseded by 0017 | R-PLAT-01 |
|
||||
| [0002](adr-0002-cni-cilium.md) | CNI & network policy & ingress: Cilium + Gateway API | Accepted | R-NET-01..07, R-DNS-01 |
|
||||
| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted | R-STATE-01..04 |
|
||||
| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted — backup and bulk-backing clauses superseded by 0018 | R-STATE-01..04 |
|
||||
| [0004](adr-0004-gitops-flux.md) | GitOps engine: Flux CD | Accepted | R-GITOPS-01..03 |
|
||||
| [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 |
|
||||
| [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 |
|
||||
| [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 |
|
||||
| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | ~~Superseded by 0012~~ | R-NET-05/06 |
|
||||
| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 |
|
||||
| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | ~~Superseded by 0023~~ | R-UPD-01..03 |
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
|
||||
| [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | ~~Superseded by 0013~~ | R-NET-05/06/07 |
|
||||
| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 |
|
||||
| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 |
|
||||
| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 |
|
||||
| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Proposed | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 |
|
||||
| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted — forward-auth enforcement point and app list superseded by 0022 | R-IDP-01..06 |
|
||||
| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted — Kill-switch and Resilience bullets superseded by 0020 | R-NET-05/06/07 |
|
||||
| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Accepted | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 |
|
||||
| [0017](adr-0017-single-environment-bare-metal-production.md) | Single environment: bare-metal production; dev retired | Accepted | R-PLAT-01, R-ENV-01/02/03, R-STATE-01/02 |
|
||||
| [0018](adr-0018-storage-backup-truenas-r2.md) | Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2 | Accepted | R-STATE-03, R-STATE-04 |
|
||||
| [0019](adr-0019-postgresql-cloudnativepg.md) | PostgreSQL via CloudNativePG | Accepted | R-IDP-05, R-STATE-01, R-STATE-02, R-STATE-03 |
|
||||
| [0020](adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) | VPN egress: single gateway node, Cilium-enforced fail-closed | Accepted | R-NET-05, R-NET-06 |
|
||||
| [0021](adr-0021-inbound-remote-access-wireguard.md) | Inbound remote access: WireGuard on a hostNetwork DaemonSet | Accepted | remote operator access (no dedicated requirement ID); bounds R-NET-01, R-SEC-02, R-SEC-04 |
|
||||
| [0022](adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | Forward-auth via per-app oauth2-proxy sidecars | Accepted | R-IDP-02, R-IDP-06 |
|
||||
| [0023](adr-0023-automated-updates-renovate.md) | Automated updates: Renovate for images and charts, every change PR-gated | Accepted | R-UPD-01, R-UPD-02, R-UPD-03, R-TEST-03 |
|
||||
| [0024](adr-0024-delivery-authority-ci-gate.md) | Delivery authority: the CI gate is the approver; sensitive-path review; attributed admin override | Accepted | R-TEST-03, R-TEST-06, R-GITOPS-01, R-SEC-02 |
|
||||
| [0025](adr-0025-in-cluster-ai-program-manager.md) | In-cluster AI Program Manager (pm-agent) | Accepted | R-GITOPS-01, R-TEST-03, R-TEST-06, R-SEC-02 |
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0001 — Kubernetes distribution: k3s
|
||||
- Status: Accepted
|
||||
- Status: Accepted — k3s, embedded etcd and its disables stand (dev provisioning clause superseded by [ADR-0017](adr-0017-single-environment-bare-metal-production.md))
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-PLAT-01, R-PLAT-02, R-ENV-01/02/03
|
||||
- Supersedes: —
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)
|
||||
- Status: Accepted
|
||||
- Status: Accepted — Longhorn stands (backup and bulk-backing clauses superseded by [ADR-0018](adr-0018-storage-backup-truenas-r2.md))
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-STATE-01, R-STATE-02, R-STATE-03, R-STATE-04
|
||||
- Supersedes: —
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0009 — Automated updates: Flux image automation + Renovate
|
||||
- Status: Accepted
|
||||
- Status: **Superseded by [ADR-0023](adr-0023-automated-updates-renovate.md)** (2026-11-08)
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-UPD-01, R-UPD-02, R-UPD-03
|
||||
- Supersedes: —
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0014 — Identity Provider: Keycloak (SSO via OIDC + forward-auth)
|
||||
- Status: Accepted
|
||||
- Status: Accepted — Keycloak, native OIDC, groups and MFA stand (forward-auth enforcement point and app list superseded by [ADR-0022](adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md))
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-IDP-01, R-IDP-02, R-IDP-03, R-IDP-04, R-IDP-05, R-IDP-06
|
||||
- Supersedes: —
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)
|
||||
- Status: Accepted
|
||||
- Status: Accepted — tunnel placement, steering and the four invariants stand (Kill-switch and Resilience bullets, and the mechanism named in invariant 3, superseded by [ADR-0020](adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md))
|
||||
- Date: 2026-08-23
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes (egress-steering + tunnel placement of): [ADR-0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding
|
||||
- Status: Proposed (decided by the operator 2026-11-08; set Accepted on merge)
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
|
||||
- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward
|
||||
only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md)
|
||||
@@ -28,9 +29,10 @@ public web apps has to come from the public answer itself working from the LAN.
|
||||
|
||||
- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only**
|
||||
connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
|
||||
- **Public records** for `exposure: public` services are **proxied CNAMEs to
|
||||
`<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service intent
|
||||
(R-DNS-02). No record is hand-edited.
|
||||
- **Public records** for services declared public (catalog `exposure: public`, or a registered
|
||||
platform route: `auth`, Grafana, Crafty's web UI, the apex and its `/runbooks`) are **proxied
|
||||
CNAMEs to `<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service
|
||||
intent (R-DNS-02). No record is hand-edited.
|
||||
- **The edge router's 80/443 forwards are closed** after cutover.
|
||||
- **Hostnames are unchanged:** `<service>.home.keithjackson.dev` and the apex
|
||||
`home.keithjackson.dev` (R-DNS-01, R-C2-01).
|
||||
@@ -40,19 +42,23 @@ public web apps has to come from the public answer itself working from the LAN.
|
||||
- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006).
|
||||
cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using
|
||||
the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used).
|
||||
- **Only `exposure: public` services have a tunnel route.** The tunnel's ingress rules end in a
|
||||
**catch-all `http_status:404`**, so any other hostname presented at the tunnel is refused
|
||||
(R-DNS-05).
|
||||
- **The tunnel's ingress rules are a deny-list plus a zone wildcard.** Each LAN/VPN-only hostname
|
||||
maps to `http_status:404`; the apex and `*.home.keithjackson.dev` then go to the Gateway; a
|
||||
**catch-all `http_status:404`** closes the list for hosts outside the zone (R-DNS-05). A static
|
||||
conformance check keeps the deny-list and the set of unpublished routes identical in both
|
||||
directions.
|
||||
- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other
|
||||
path. It is LAN/VPN-only (R-SEC-04).
|
||||
|
||||
**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent,
|
||||
radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services
|
||||
**unifi** and **crafty-mc** (Minecraft). They get no public DNS record and no tunnel route.
|
||||
**unifi** and **crafty-mc** (Minecraft). The tunnel refuses all of them with a 404. None has a
|
||||
public DNS record, except that unifi and crafty-mc hold **DNS-only records at their LAN addresses**,
|
||||
which are never proxied or tunnelled.
|
||||
|
||||
**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming
|
||||
keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes
|
||||
**LAN-only**: no tunnel route and no proxied record.
|
||||
**LAN-only**: refused at the tunnel, with no proxied record.
|
||||
|
||||
**Unchanged:**
|
||||
- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record,
|
||||
@@ -88,7 +94,7 @@ keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.
|
||||
## Consequences
|
||||
- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT
|
||||
hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit,
|
||||
declared, per-hostname act (tunnel route + record), and anything else 404s at the edge.
|
||||
declared, per-hostname act (a public record), and LAN/VPN-only names 404 at the tunnel.
|
||||
- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps
|
||||
working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the
|
||||
internal-only names. Public apps without a local record are unreachable from the LAN during such an
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# ADR-0017 — Single environment: bare-metal production; dev retired
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-PLAT-01, R-ENV-01/02/03, R-STATE-01/02
|
||||
- Supersedes: ADR-0001 provisioning clause only ("Nodes are provisioned by OpenTofu (libvirt, dev) /
|
||||
bare-metal imaging (prod) + cloud-init, and joined/configured by Ansible"). The rest of ADR-0001
|
||||
stands: k3s, 3-node HA on embedded etcd, and the bundled components disabled at install.
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
|
||||
## Context
|
||||
ADR-0001 and the original specs assumed two environments: a dev/test cluster of local libvirt KVMs,
|
||||
provisioned by OpenTofu and cloud-init, and a prod cluster of 3 mini-PCs, kept design-only until a
|
||||
later phase.
|
||||
|
||||
Delivery did not stay on that path. The workstation host ran out of RAM and disk, so dev moved to VMs
|
||||
on a Proxmox host. The operator opened Phase 7 on 2026-08-31, and prod was built on 3 HP EliteDesk
|
||||
Mini bare-metal nodes. On 2026-09-12/13 prod took over the apex `home.keithjackson.dev`, and the dev
|
||||
VMs were stopped. The storage pools of the host those VMs ran on then moved to a standalone NAS, and
|
||||
the migration plan retires that host.
|
||||
|
||||
Keeping "dev/test" in the specification now describes something that does not exist. It also leaves
|
||||
R-ENV-01 and R-ENV-03 unsatisfiable as written.
|
||||
|
||||
## Decision
|
||||
The platform has **one live environment: prod, 3× mini-PC bare metal**.
|
||||
|
||||
- **Create** is an imaged, checklist-driven operator step: there is no hypervisor API to call on
|
||||
bare metal. The checklist is `infra/PROD-PROVISIONING.md` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
- **Baseline, cluster-join and manage** are code: Ansible owns all of them, including the first-boot
|
||||
baseline that cloud-init used to carry. Cloud-init is not part of node provisioning.
|
||||
- OpenTofu is not part of node provisioning.
|
||||
- The Kubernetes API has one HA endpoint, a **kube-vip** virtual IP across the 3 servers. The address
|
||||
a joining node uses during a cold bootstrap is deliberately a different one, because the virtual IP
|
||||
does not exist until a cluster does.
|
||||
- GitOps configuration stays **one base plus an environment-scoped overlay**. No second environment
|
||||
exists, but the separation is kept, so a future environment needs overrides only.
|
||||
- With no staging surface, the safeguard is the merge gate: every change reaches prod through the PR
|
||||
gate and Flux. Destructive or fault-injecting tests against prod are deliberate, scheduled
|
||||
exercises.
|
||||
|
||||
## Alternatives considered
|
||||
- **Keep a dev cluster on Proxmox.** Rejected: the migration plan retires that host, and dev does
|
||||
not move to the NAS, which is storage only.
|
||||
- **Rebuild dev as local KVMs on the workstation.** Rejected: that is the path already abandoned for
|
||||
lack of RAM and disk.
|
||||
- **Drive bare metal through Cluster API or PXE automation.** Deferred: imaging plus Ansible is
|
||||
sufficient and lower-complexity for 3 nodes.
|
||||
|
||||
## Consequences
|
||||
- The spec stops gating work on a dev cluster; R-ENV-01/02/03 and R-PLAT-01 are reworded to match.
|
||||
- Conformance runs against prod. There is no separate cluster on which to absorb a bad change, so the
|
||||
static gate and the discipline around fault injection carry more weight.
|
||||
- Node-rebuild is exercised by hand rather than by a destroy/rebuild pipeline; a node is reimaged and
|
||||
rejoined through the checklist and Ansible.
|
||||
- The `clusters/dev` overlay, `infra/tofu` and dev defaults remaining in the implementation repo are
|
||||
inert. Removing them is implementation cleanup, and a conformance check that pins them must change
|
||||
with them.
|
||||
- ADR-0001's note that prod must not go below 3 servers stands, and it has no dev counterpart.
|
||||
@@ -0,0 +1,72 @@
|
||||
# ADR-0018 — Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2; no Longhorn backup target
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
- Serves: R-STATE-03, R-STATE-04
|
||||
- Supersedes: [ADR-0003](adr-0003-storage-longhorn.md) — the backup and bulk-backing clauses only.
|
||||
**Longhorn as the replicated block store, 3 replicas by default, and Plex's metadata on Longhorn
|
||||
all stand.**
|
||||
|
||||
## Context
|
||||
ADR-0003 left two things open or conditional. The bulk tier's backing was "decided under OQ-2", and
|
||||
the DR path was "Velero (with Longhorn backups)" to an unnamed off-cluster target (OQ-BKP).
|
||||
|
||||
Operating the platform settled both:
|
||||
- The S3 endpoint on the old Proxmox host could not serve prod. It was unreachable across the network
|
||||
boundary, measured from a pod inside prod, and that host shared a failure domain with the VMs it
|
||||
ran. With WAL archiving on and no working target, Postgres refused to recycle WAL and database
|
||||
volumes filled, so the missing target was an availability incident as well as a DR gap.
|
||||
- A Longhorn backup target was escalated as a blocker for the DR drill. The owner ruled on 2026-09-13
|
||||
that Velero + kopia to R2 is the volume backup and that no Longhorn target will be configured.
|
||||
- The Proxmox host was decommissioned and its ZFS pools physically moved to a storage-only TrueNAS
|
||||
appliance on the nodes' own LAN. The multi-terabyte library was already there, so the tier attaches
|
||||
existing data and creates none.
|
||||
|
||||
Uniform file-system backup also collides with NFS. Opt-out backup would read the whole media library
|
||||
over NFS every night and upload it, once per consuming namespace.
|
||||
|
||||
## Decision
|
||||
**Bulk tier.** Bulk media lives on a standalone **TrueNAS** appliance, storage only, consumed over
|
||||
**NFS**. A catalog `class: bulk` entry with a named export renders a statically bound volume, and the
|
||||
server address is a per-environment value. There is no dynamic provisioning and no NFS CSI driver.
|
||||
|
||||
**Off-cluster target.** **Cloudflare R2**, one bucket, with Velero and CloudNativePG/Barman under
|
||||
disjoint prefixes so that neither's retention can reach the other's objects.
|
||||
|
||||
**Volume and resource backup.** **Velero with kopia file-system backup**, opt-out by default so no
|
||||
volume is forgotten. No CSI snapshot data-mover and no Volume Snapshot Location.
|
||||
|
||||
**No Longhorn backup target.** Longhorn snapshots remain an in-cluster mechanism only.
|
||||
|
||||
**Database backup.** Continuous WAL and base backups through CloudNativePG/Barman ([ADR-0019](adr-0019-postgresql-cloudnativepg.md)).
|
||||
|
||||
**Exclusions by property.** A volume policy skips NFS and emptyDir volumes by source type, so the rule
|
||||
covers volumes that do not exist yet. The bulk tier is therefore **outside platform backup**.
|
||||
Reconstructable data (monitoring's metrics and logs, the UniFi VM disk) is captured as Kubernetes
|
||||
resources only.
|
||||
|
||||
## Alternatives considered
|
||||
- **MinIO/versitygw on the Proxmox host** — unreachable from prod and in the same failure domain as
|
||||
what it protected; the host no longer exists.
|
||||
- **MinIO on the TrueNAS** — planned during the host migration; not adopted. It would keep backups on
|
||||
site beside the cluster, where R2 is off-site with free egress, so restore rehearsals carry no cost.
|
||||
- **Longhorn backup target as well as, or instead of, Velero** — a second restore path, declined by
|
||||
the owner because kopia to R2 already provides a genuine off-cluster copy.
|
||||
- **Excluding media namespaces from Velero** — rejected: those namespaces also hold irreplaceable
|
||||
Longhorn PVCs.
|
||||
- **Per-pod exclusion annotations on NFS volumes** — rejected: fails open. The next NFS volume is
|
||||
uploaded in full until someone remembers to annotate it.
|
||||
- **Dynamic NFS provisioning (the `bulk` StorageClass ADR-0003 implied)** — unnecessary: every bulk
|
||||
volume attaches a dataset that already holds data.
|
||||
|
||||
## Consequences
|
||||
- The restore path is single. A failure in Velero's path to R2 has no Longhorn-side fallback; revisit
|
||||
if that path proves unreliable, not on principle.
|
||||
- R2 is not a full S3 implementation and rejects some behaviours other stores tolerate. Compatibility
|
||||
settings live in the implementation repo.
|
||||
- A bound NFS volume's source is immutable, so moving the bulk server means recreating the volumes
|
||||
rather than reconciling them.
|
||||
- The bulk tier's durability is the NAS's own. Whether any of it is irreplaceable, notably photos, is
|
||||
an open operator decision.
|
||||
- Detail: `docs/BULK-NFS-STORAGE.md` and `platform/velero/` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
@@ -0,0 +1,50 @@
|
||||
# ADR-0019 — PostgreSQL via CloudNativePG
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
- Serves: R-IDP-05, R-STATE-01, R-STATE-02, R-STATE-03
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
[ADR-0014](adr-0014-idp-keycloak.md) requires Keycloak's PostgreSQL to live on replicated Longhorn,
|
||||
be DR-critical and be backed up off-cluster; it does not say how PostgreSQL is run. Nextcloud and
|
||||
Semaphore need PostgreSQL too.
|
||||
|
||||
The first design used the Bitnami Keycloak chart's bundled PostgreSQL. Bitnami's free images were
|
||||
purged or relocated, so a deploy on stock tags fails to pull.
|
||||
|
||||
Single-instance PostgreSQL also proved insufficient on its own terms. Longhorn replication preserves
|
||||
the data through a node loss but not the service. With one instance each, all three primaries drifted
|
||||
onto the same node, so one node failure would have taken Nextcloud, Semaphore and SSO down together.
|
||||
|
||||
## Decision
|
||||
Run every platform PostgreSQL database under the **CloudNativePG** operator, one cluster per database:
|
||||
- **Three instances** with hard anti-affinity, one per node, each on the replicated `longhorn` class.
|
||||
- **Off-cluster recovery** through Barman: continuous WAL archiving plus a daily base backup to the
|
||||
object store ([ADR-0018](adr-0018-storage-backup-truenas-r2.md)), retained 30 days.
|
||||
- Primary changes roll by **switchover** rather than an in-place restart.
|
||||
- Standbys are added only behind a working archive, because they replicate corruption and are not a
|
||||
backup.
|
||||
|
||||
Scope: services that need PostgreSQL. An application with a small embedded datastore (Gitea's SQLite,
|
||||
for example) keeps it on its Longhorn PVC.
|
||||
|
||||
## Alternatives considered
|
||||
- **Bitnami-bundled PostgreSQL** — images no longer dependably available from the free registry.
|
||||
- **Single-instance PostgreSQL on Longhorn** — keeps the data through a node loss but loses the
|
||||
service until the pod reschedules, and gives no point-in-time recovery by itself.
|
||||
- **Relying on file-system volume backup alone** — no point-in-time recovery; a kopia copy of a live
|
||||
data directory is the weaker path. It is retained as a secondary copy, not as the database backup.
|
||||
- **PostgreSQL embedded in each application's chart** — one lifecycle per application, with backup
|
||||
and failover solved separately each time.
|
||||
|
||||
## Consequences
|
||||
- Failover takes seconds, and every database has one declarative point-in-time recovery path.
|
||||
- WAL archiving is an availability dependency. If archiving fails, PostgreSQL correctly refuses to
|
||||
recycle WAL and the volume fills, so archive health is alerted on as an availability signal.
|
||||
- Storage cost multiplies: three instances on three-way replicated volumes means nine copies of each
|
||||
database. Accepted, because the databases are small.
|
||||
- The operator's custom resources need CRDs before they apply, which fits the platform's
|
||||
controllers-then-configs ordering.
|
||||
- Detail: `platform/cnpg/` and the database manifests in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
@@ -0,0 +1,61 @@
|
||||
# ADR-0020 — VPN egress: single gateway node, Cilium-enforced fail-closed
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
- Serves: R-NET-05, R-NET-06
|
||||
- Supersedes (the **Kill-switch** and **Resilience** bullets of): [ADR-0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md)
|
||||
— whose node-level tunnel placement, Cilium Egress Gateway steering and four invariants stand.
|
||||
|
||||
## Context
|
||||
ADR-0015 made a host nftables rule the fail-closed backstop for `vpn`-class traffic, and relied on two
|
||||
gateway nodes plus "Cilium Egress Gateway HA" for failover. Building and measuring it disproved both.
|
||||
|
||||
- **nftables never sees the traffic.** Egress Gateway requires eBPF masquerade, and the datapath
|
||||
redirects steered packets straight to the egress device without passing netfilter. A counters-only
|
||||
table on the gateway counted zero steered packets on the forward and postrouting hooks while a
|
||||
`vpn`-class pod egressed, with the same hooks counting WAN and node-originated traffic.
|
||||
- **Cilium is what fails closed.** With the tunnel stopped and during a restart, packet captures on
|
||||
both the WAN and the tunnel device showed nothing leaving either one.
|
||||
- **Gateway HA does not exist in Cilium OSS.** The policy's `egressGateway` field resolves a matching
|
||||
set to the first node in name order. The plural `egressGateways` field shards endpoints across nodes
|
||||
rather than providing standby: losing a node blackholes its shard and re-shards the rest, and each
|
||||
shard exits from a different public IP. Upstream: cilium/cilium#30157 and #18230.
|
||||
- **Cilium caches the tunnel interface.** It resolves the interface once. A reconnect re-creates the
|
||||
device under a new index, and `vpn`-class egress is then silently blackholed.
|
||||
|
||||
## Decision
|
||||
- **Exactly one egress-gateway node.** One node carries the gateway label. There is no failover, and
|
||||
a second labeled node is never added as a "standby".
|
||||
- **Fail-closed is enforced by Cilium's egress datapath.** R-NET-06 is unchanged: tunnel down means no
|
||||
`vpn`-class packet reaches the plain internet. The guarantee is proven by the `vpn-killswitch`
|
||||
conformance test, against Cilium's behaviour.
|
||||
- **The host nftables table is a narrow backstop.** It stays, with counters, for what it can see:
|
||||
node-originated traffic sourced from a tunnel address.
|
||||
- **The gateway node signals every reconnect to Cilium**, through a node label that makes Cilium
|
||||
re-resolve the interface. A node never writes to the Flux-managed policy.
|
||||
|
||||
### What stands from ADR-0015
|
||||
Tunnel placement (host OpenVPN service, split-tunnel, Ansible-managed), steering (a
|
||||
`CiliumEgressGatewayPolicy` selecting `egress-class=vpn`), host-side credentials, and all four
|
||||
invariants. Invariant 3's scope stands: only `vpn`-class traffic is dropped when the tunnel is down.
|
||||
The enforcing mechanism it names is the one this ADR replaces.
|
||||
|
||||
## Alternatives considered
|
||||
- **Label a second node as standby** — rejected: it carries nothing as a standby, and because selection
|
||||
is by name order it can silently take the role, the tunnel and the public egress IP.
|
||||
- **Plural `egressGateways`** — rejected: sharding, not failover; a node loss still blackholes a share,
|
||||
breaks live connections, and splits the service across exit IPs.
|
||||
- **Make the Cilium policy stamp a mark for a host firewall to match** — not implementable: Cilium OSS
|
||||
has no supported way to mark this traffic, and it would not reach a host firewall hook anyway.
|
||||
- **Keep ADR-0015's text and treat nftables as authoritative** — rejected: it would document a control
|
||||
that measurably does nothing for the traffic it claims to protect.
|
||||
|
||||
## Consequences
|
||||
- The gateway node is a **single point of failure for `vpn`-class egress**. Its loss is an outage with
|
||||
no leak, the direction R-NET-06 requires.
|
||||
- **Labeling is a safety-relevant act.** Labeling another node moves the tunnel and the public egress
|
||||
IP silently, with no event or alert.
|
||||
- The re-steer path is alerted (a reconnect not reflected in the label; a gateway with no label).
|
||||
Cilium's egress map is not exposed as a metric, so "the label changed and Cilium ignored it" is
|
||||
not detected by these alerts.
|
||||
- If Cilium OSS gains real egress-gateway failover, this ADR can be reopened by a superseding ADR.
|
||||
@@ -0,0 +1,67 @@
|
||||
# ADR-0021 — Inbound remote access: WireGuard on a hostNetwork DaemonSet
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
- Serves: remote operator access (no dedicated requirement ID); bounds R-NET-01, R-SEC-02, R-SEC-04
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
The operator's Android, Linux, Windows and other devices need to reach the home network from anywhere
|
||||
as though on the LAN. Several services are LAN/VPN-only by design (ADR-0016), and the Kubernetes API is
|
||||
LAN/VPN-only (R-SEC-04), so the "VPN" in that phrase needs a defined mechanism. This is unrelated to
|
||||
`vpn`-class egress (ADR-0015, ADR-0020), which sends pods out through a provider tunnel.
|
||||
|
||||
Two facts about this cluster shape the design:
|
||||
- Cilium decides what a packet may reach from the **security identity of its source**. Measured: a
|
||||
request to the shared Gateway from inside a pod is refused for every Gateway hostname. A
|
||||
pod-networked VPN server would give every client that pod's identity: a healthy tunnel on which
|
||||
every web UI is refused.
|
||||
- The SOHO edge has no automation and cannot hairpin NAT. An inbound forward is a manual step that names
|
||||
one node address.
|
||||
|
||||
## Decision
|
||||
**Remote access is WireGuard on udp/51820, run as a Flux-managed DaemonSet in the host network
|
||||
namespace on every node.**
|
||||
|
||||
- **Host network** so that clients never carry a pod identity.
|
||||
- **DaemonSet** so that every node holds an identical warm listener, and losing the forwarded node is a
|
||||
one-line router change with no cluster work and no new client profiles.
|
||||
- **Authorization is a reviewable git diff:** peer public keys are in plain text in git, so granting or
|
||||
revoking a device is a reviewed change that rolls the listeners. The server's private key is
|
||||
SOPS-encrypted (ADR-0007), and joined with the peer list only in memory at pod start.
|
||||
- **Reach equals a LAN machine:** the whole LAN, the Gateway, Pi-hole, and the internet through the
|
||||
house WAN. Clients are `internet` class, never steered to the provider tunnel.
|
||||
- **Endpoint:** the DNS-only `wan.` record (ADR-0016). A client inside the house cannot use it, as there
|
||||
is no hairpin, and uses a node's LAN address instead.
|
||||
|
||||
**This is a deliberate exception to default-deny (R-NET-01).** A host-network workload has no Cilium
|
||||
endpoint, so no network policy applies to it. The exception is scoped to this platform component;
|
||||
application namespaces remain barred from the host network by admission policy. The boundary is
|
||||
WireGuard's peer authentication: a packet from an unknown key gets no reply, so the port is
|
||||
indistinguishable from a closed one.
|
||||
|
||||
## Alternatives considered
|
||||
- **OpenVPN** — rejected: a CA, per-device certificates and a revocation list are long-lived state to
|
||||
protect and rotate.
|
||||
- **IKEv2/IPsec** — rejected: painful to containerize and brittle behind double NAT.
|
||||
- **Pod-networked WireGuard** — rejected: clients inherit the pod's identity and every Gateway hostname
|
||||
refuses them, while the tunnel reports healthy.
|
||||
- **A LoadBalancer VIP for the listener** — rejected: adds an L2-announcement lease, SNAT and a
|
||||
node-to-node hop to the one path whose failure means the operator is locked out while away.
|
||||
- **Single-replica Deployment** — rejected: a reschedule silently invalidates the router forward.
|
||||
- **Carry it over the Cloudflare Tunnel** — not possible: the tunnel carries HTTP(S), and WireGuard is
|
||||
UDP to a node.
|
||||
- **Encrypt the peer list / add preshared keys** — rejected: hides the change a reviewer most needs to
|
||||
see, and a preshared key would be a further secret for a threat not in the model.
|
||||
|
||||
## Consequences
|
||||
- **No network policy constrains this workload.** Peer authentication and the single forwarded port are
|
||||
the controls.
|
||||
- **An enrolled device is as privileged as a machine in the house**, including reach to the nodes'
|
||||
control-plane ports. A stolen device is that device until revoked, so revocation must be prompt.
|
||||
Narrowing is possible per profile (split tunnel) or by host firewall rules in the listener.
|
||||
- The WAN IP stays public for this port (ADR-0016); the forward is a manual edge step.
|
||||
- The tunnel carries IPv4 only. Client profiles default to a conservative MTU, because a client's own
|
||||
path can be narrower than the server's.
|
||||
- No per-peer metrics exist; a WireGuard exporter would need a second privileged container beside the
|
||||
internet-facing tunnel.
|
||||
@@ -0,0 +1,69 @@
|
||||
# ADR-0022 — Forward-auth via per-app oauth2-proxy sidecars
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
- Serves: R-IDP-02, R-IDP-06
|
||||
- Supersedes: [ADR-0014](adr-0014-idp-keycloak.md) — **enforcement point and app list only**: the
|
||||
"at the Gateway" clause and the list of forward-auth apps in its Decision. Everything else in
|
||||
ADR-0014 stands.
|
||||
|
||||
## Context
|
||||
ADR-0014 put a single oauth2-proxy at the Gateway as an external-authorization check in front of
|
||||
every app without usable OIDC. That design needs a per-route authorization hook on the shared
|
||||
Gateway, and the platform's Gateway implementation does not offer a usable one:
|
||||
|
||||
- Cilium generates and owns the shared Gateway's Envoy configuration, regenerating it on every
|
||||
route change, with no supported way to inject an authorization filter into it.
|
||||
- Cilium's native Gateway API `ExternalAuth` filter needs the **experimental** Gateway API CRD
|
||||
channel, a cluster-wide change, and shipped with a fail-**open** bug in the very property
|
||||
R-IDP-02 forbids. It was evaluated and deferred.
|
||||
- Without a gate, the bare UIs stayed reachable unauthenticated on the LAN.
|
||||
|
||||
Crafty's web UI was already gated by a co-located oauth2-proxy, which proved the pattern on this
|
||||
platform. Detail: `docs/plans/cilium-native-externalauth-evaluation.md` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
|
||||
## Decision
|
||||
Forward-auth is enforced by a **per-app oauth2-proxy running as a reverse-proxy sidecar in the
|
||||
app's pod**, in the data path, rather than by a shared check at the Gateway.
|
||||
|
||||
- The app's route targets the sidecar's port. Network policy opens **only** that port to the
|
||||
Gateway, so the bare UI port is unreachable from it and a failed or absent gate fails closed.
|
||||
- Each gate is its own instance: its own IdP client, its own SOPS-encrypted credentials, and the
|
||||
authorized groups declared in the service's `auth` block.
|
||||
- **Session cookies are per-instance**: a unique cookie name per gate, scoped to the app's own
|
||||
host. Instances hold different cookie secrets, so a shared name on a zone-wide domain makes them
|
||||
evict one another's sessions.
|
||||
- An app MAY declare the path prefixes of its programmatic API, so unauthenticated calls there are
|
||||
refused rather than redirected to an HTML login page.
|
||||
- The catalog renderer generates the sidecar for every `forward-auth` service with an exposed UI.
|
||||
|
||||
**App list (replaces ADR-0014's).** Gated this way: PiHole, Sonarr, Radarr, Jackett, qBittorrent
|
||||
and the Crafty web UI. Homepage has no usable OIDC and requires a gate under R-IDP-02.
|
||||
Prometheus, Alertmanager and Hubble UI are not exposed, so no gate applies; any future exposure
|
||||
MUST be gated.
|
||||
|
||||
## Alternatives considered
|
||||
- **Self-authored Envoy listener with an authorization filter on its own VIP** — works, and was
|
||||
proven against the live proxy, but needs hand-maintained TLS and routes per app, a second ingress
|
||||
address and DNS repointing. Rejected; retained in the implementation repo as an unwired template.
|
||||
- **Cilium native `ExternalAuth`** — the cleanest shape and the long-term target, one per-route
|
||||
filter and no per-app proxies. Deferred: experimental CRDs, an immature security-critical filter,
|
||||
thin documentation. A future ADR may adopt it and retire the sidecars.
|
||||
|
||||
## Consequences
|
||||
- Enforcement is Gateway-API-native (a backend swap) and does not depend on the Gateway's generated
|
||||
configuration.
|
||||
- More moving parts: one proxy, one IdP client and one credential set per gated app. Adding a gated
|
||||
app needs all three.
|
||||
- Sessions are per-host. A live Keycloak session makes the bounce silent, so SSO still holds, but a
|
||||
browser performs one redirect per app.
|
||||
- An expired session answers an unauthenticated XHR with a redirect that resolves as a successful
|
||||
HTML response, which is why API paths are declarable and why each gate exposes metrics for
|
||||
scraping ([10](../docs/10-observability-and-alerting.md)).
|
||||
- Crafty is platform-managed outside the renderer, so its gate is hand-maintained and must follow
|
||||
the same per-instance cookie rule.
|
||||
- A UI whose pod the renderer does not generate, such as Homepage, gets a gate only by explicit
|
||||
work ([known gaps](../docs/16-known-gaps.md)).
|
||||
- [15](../docs/15-identity-and-access.md) carries the resulting design; ADR-0014's text is not
|
||||
edited.
|
||||
@@ -0,0 +1,64 @@
|
||||
# ADR-0023 — Automated updates: Renovate for images and charts, every change PR-gated
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-UPD-01, R-UPD-02, R-UPD-03, R-TEST-03
|
||||
- Supersedes: [ADR-0009](adr-0009-automated-image-updates.md) (the VM guest mechanism stands and is
|
||||
restated below)
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
|
||||
## Context
|
||||
ADR-0009 split updates by artifact: Flux image-update-automation for container image tags, Renovate
|
||||
for charts and manifests. The image half could not be delivered on this platform's terms:
|
||||
|
||||
1. **It bypasses the merge gate.** Image automation commits and pushes rather than opening a pull
|
||||
request. Aimed at `main` it writes past the PR flow; aimed at a side branch, nothing closes the
|
||||
loop.
|
||||
2. **It cannot write here.** `main` is protected against direct pushes for every account, and Flux's
|
||||
source is anonymous and read-only. Enabling write-back means an in-cluster git **write** credential
|
||||
for the repository that defines the cluster, a far larger blast radius than a CI-scoped token.
|
||||
3. **Renovate already opens PRs**, so every change it proposes goes through the gate with no extra
|
||||
machinery.
|
||||
|
||||
The controller was authored but never enabled, and no image-policy objects ever existed in the
|
||||
cluster. Meanwhile plain `image:` tags were maintained by nothing at all, and pins drifted many minor
|
||||
versions behind. This ADR closes a path that was never taken and changes no running state.
|
||||
|
||||
## Decision
|
||||
**Renovate owns container images, Helm charts and manifest versions.** It runs on a schedule as a
|
||||
Gitea Actions job and every change it proposes is a machine-authored pull request through the merge
|
||||
gate.
|
||||
|
||||
- **Minor, patch and digest** updates auto-merge on a green gate.
|
||||
- **Majors never auto-merge**; they wait for human review of the machine-authored PR.
|
||||
- **Operator-designated blast-radius components never auto-merge**, not even a patch, because a green
|
||||
static gate says nothing about whether a bump breaks them, or because they are the merge gate
|
||||
itself. The list and each reason are data in the Renovate configuration.
|
||||
- **The platform's own CI-built images** are pinned by digest and follow a moving tag through a
|
||||
marker comment; Renovate proposes the digest bump.
|
||||
- **Per-service policy** is declared in git. Renovate does not read the ServiceDefinition, so
|
||||
ceilings and cadences take effect through per-package rules in its configuration.
|
||||
- **VM guests → in-guest auto-update**, declared in cloud-init, unchanged from ADR-0009.
|
||||
|
||||
Flux image-update-automation stays disabled. Flux itself stands
|
||||
([ADR-0004](adr-0004-gitops-flux.md)); its native image automation is no longer a load-bearing part
|
||||
of that choice, and native SOPS decryption and reconciliation remain so.
|
||||
|
||||
## Alternatives considered
|
||||
- **Flux image automation writing to `main`** — bypasses the required gate; needs an in-cluster write
|
||||
credential. Rejected.
|
||||
- **Flux image automation writing to a side branch** — nothing opens or merges the PR; bumps accumulate
|
||||
unseen. Rejected.
|
||||
- **Auto-merge everything on green** — the gate is static for these changes and cannot see whether a
|
||||
bump breaks the storage plane, the IdP or the gate itself. Rejected for the blast-radius set.
|
||||
- **Digest-tracking a mutable stable tag** — loses version visibility and semver guardrails; ADR-0009
|
||||
already held this as an escape hatch only. Not adopted.
|
||||
|
||||
## Consequences
|
||||
- Every update is auditable as a PR and is held to the same gate as a human change (R-TEST-03).
|
||||
- R-UPD-02 is amended to match: allowed classes are hands-off; majors and blast-radius components MAY
|
||||
wait for human review of the machine-authored change. No human or AI authors a version bump.
|
||||
- Update-failure signals and verification written around the Flux controller must move to Renovate's
|
||||
path ([known gaps](../docs/16-known-gaps.md)).
|
||||
- Renovate is a CI job with a scoped token; its own version never auto-merges, because nothing
|
||||
re-validates its configuration against a new release before it takes effect.
|
||||
- The update cadence is bounded by CI capacity, so PR concurrency and hourly rates are capped.
|
||||
@@ -0,0 +1,55 @@
|
||||
# ADR-0024 — Delivery authority: the CI gate is the approver; sensitive-path review; attributed admin override
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-TEST-03, R-TEST-06, R-GITOPS-01, R-SEC-02
|
||||
- Supersedes: —
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
|
||||
## Context
|
||||
[ADR-0010](adr-0010-conformance-testing.md) makes the conformance suite gate merges but does not say
|
||||
who approves. The operator's stated principle (2026-09-29): the CI/CD pipeline is the approver, and
|
||||
if it is not good enough to be trusted for that, it is not good enough to do its job and needs
|
||||
improving. Manual approval belongs only on architecturally or security-relevant changes.
|
||||
|
||||
That has one structural hole: **a change to the gate cannot be gated by the gate**, since a PR that
|
||||
weakens a check would be approved by the weakened check.
|
||||
|
||||
There is also one human on this platform, and Gitea refuses to let an author approve their own pull
|
||||
request. Requiring one approval on the branch was a lock-out rather than a protection: a change with a
|
||||
green gate and no possible reviewer could never merge, including the change that would fix that.
|
||||
|
||||
## Decision
|
||||
- **The merge gate is the approver.** It is the single required status check on `main`. Branch
|
||||
protection requires zero approvals, direct pushes are rejected, and **a red gate is unmergeable by
|
||||
everyone, administrators included**.
|
||||
- **Sensitive paths need a non-author approving review.** This covers the pipeline, the tests and
|
||||
harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the RBAC, and the
|
||||
delivery contract. The path list is data with a stated reason per entry, and is itself sensitive. The
|
||||
stage is **fail-closed**: if it cannot evaluate, it blocks, since a skip there would be a bypass.
|
||||
- **An administrator may waive that review** with an explicit directive on the PR. It must start a line
|
||||
and carry a reason. Permission is re-checked live when the gate runs. A newer commit invalidates it.
|
||||
The overrider, source, time and reason are printed in the gate output.
|
||||
- **Agents never use the override on their own work**, regardless of the permission their account
|
||||
holds. This is a rule of the delivery contract; the check itself authorises by live permission alone
|
||||
(operator decision, 2026-10-01).
|
||||
|
||||
## Alternatives considered
|
||||
- **Require one approval on the branch** — unsatisfiable with one human and author self-approval
|
||||
refused. Rejected after a green PR proved unmergeable.
|
||||
- **Allow administrators to merge past the gate** — reopens the route by which PRs had merged with zero
|
||||
checks while settings read as enforced. Rejected; only the approval count was dropped.
|
||||
- **Gate approves everything, including its own changes** — a weakening PR would approve itself.
|
||||
Rejected.
|
||||
- **No override** — the sole administrator could not change the pipeline at all. Rejected.
|
||||
- **Restrict the override to an allowlist of human accounts** — stronger, and reachable later without
|
||||
logic changes. Not adopted at this time, with the consequence below accepted knowingly.
|
||||
|
||||
## Consequences
|
||||
- Ordinary changes merge on green with no human step; a weak gate is a defect, not a reason for review.
|
||||
- For an administrator, "the gate cannot gate itself" is **advisory, not structural**. What is
|
||||
guaranteed is an attributed, reasoned, commit-scoped record of every waiver.
|
||||
- An agent account with administrator permission *can* clear the review on its own PR. Only the
|
||||
delivery contract prohibits it. Automated agents such as pm-agent
|
||||
([ADR-0025](adr-0025-in-cluster-ai-program-manager.md)) therefore use non-administrator accounts.
|
||||
- Live Gitea settings sit outside git, so a gate stage compares them against a committed declaration.
|
||||
- The approver is only as good as what the gate runs ([known gaps](../docs/16-known-gaps.md)).
|
||||
@@ -0,0 +1,52 @@
|
||||
# ADR-0025 — In-cluster AI Program Manager (pm-agent)
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-GITOPS-01, R-TEST-03, R-TEST-06, R-SEC-02
|
||||
- Supersedes: —
|
||||
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
||||
|
||||
## Context
|
||||
The operator wants a Gitea issue to be able to become a pull request without a workstation session. That
|
||||
puts an AI agent inside the cluster, holding a repository write token and acting on text written by
|
||||
whoever can reach the repository. It is a new trust boundary: input is untrusted, the actor can author
|
||||
code, and it needs egress to an external model API. "May create Jobs" is also dangerous in its own
|
||||
right, since Kubernetes places no check on a Job's `serviceAccountName`.
|
||||
|
||||
## Decision
|
||||
Run **pm-agent**: a dispatcher in its own namespace that polls labelled issues on the implementation
|
||||
repo and acts as the Program Manager on each thread; replying to an explicit mention on a PR is an
|
||||
optional capability, off by default. It starts at most a small, capped number of worker Jobs, each
|
||||
of which authors a change, runs the merge gate and opens a PR. The issue thread and the PR hold all of its state and correspondence.
|
||||
|
||||
- **Contributor, never approver.** It cannot push to `main` and never merges. Its bot account is a
|
||||
non-administrator, so it cannot waive review ([ADR-0024](adr-0024-delivery-authority-ci-gate.md)).
|
||||
- **Three independent layers between a hostile comment and the cluster.** The deciding model holds no
|
||||
tools; the dispatcher accepts only a validated vocabulary and refuses rather than repairs; and a
|
||||
fail-closed admission policy admits only the committed Job shape.
|
||||
- **Workers hold no cluster credential**; egress is limited to Gitea and the model API over TLS.
|
||||
- **Delivery tooling, not a catalog service.** It declares objects a ServiceDefinition cannot express
|
||||
and is hand-authored under `platform/` with its own non-gating Flux tier, so it can never stall the
|
||||
app tier. It adds no `R-*` requirement.
|
||||
- **Bound by the governance every agent is bound by.** It is an implementation agent under
|
||||
[AGENTS.md](../AGENTS.md) and R-TEST-06. It does not hold the System Architect role, and it declines
|
||||
and escalates work that needs a spec change.
|
||||
|
||||
## Alternatives considered
|
||||
- **Receive Gitea webhooks** — needs a security setting on the git host widened so it can call into
|
||||
the cluster, and a missed delivery is lost silently. Rejected; polling reads current state.
|
||||
- **One agent that both triages and implements** — puts tools in reach of raw issue text. Rejected.
|
||||
- **A database for state** — a second source of truth beside the thread. Rejected.
|
||||
- **Federated Keycloak machine identity** — Gitea's API cannot validate a third-party token. Rejected;
|
||||
a local account with a scoped token is what Gitea can verify.
|
||||
- **RBAC alone as the boundary** — cannot constrain what a created Job runs as. Rejected; admission is
|
||||
the boundary and RBAC its necessary half.
|
||||
|
||||
## Consequences
|
||||
- Issues become an intake for implementation work, and every result lands as an ordinary gated PR.
|
||||
- Accepted residual risks, documented in the implementation repo's threat model: a worker can read its
|
||||
own credentials (bounded by one-repository, non-admin scope and by egress); TLS egress is wider than
|
||||
one API host; and no Gitea token scope separates pushing from merging, so "never merges" rests on the
|
||||
code making no merge call.
|
||||
- An external model provider sees repository and issue content.
|
||||
- Its capability can be stopped by configuration without removing the service.
|
||||
- The worker cannot measure the live platform; anything needing a cluster stays unverified in its report.
|
||||
Reference in new issue
Block a user