Home-Network/home-network-k3s-impl

0 Fork 0

Files

T

History

jakman85 f338e91770

gate / gate (push) Has been cancelled

Details

gate / conformance (push) Has been cancelled

Details

feat(cnpg): instances 3 on all three Clusters — a standby to fail over to

All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication
saves the DATA when a node dies; it does not save the SERVICE — a single
instance has to be rescheduled, re-attach its volume and replay WAL before
Postgres accepts a connection again.

With nothing in the spec keeping them apart, all three primaries had drifted
onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node
took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl
OIDC, Grafana, Headlamp, Gitea and forward-auth.

Three instances give each Cluster a streaming standby CNPG can promote in
seconds, repointing the -rw Service with no volume re-attach and no restore.

UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate
corruption and deletions faithfully; only the Barman archive is a backup, and
adding standbys while archive_mode=on with a failing archive_command would
have multiplied the disk-full clock, not hedged it. All three Clusters on
both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real
firstRecoverabilityPoint (verified read-only, 2026-09-07).

Scaling out is non-disruptive on its own: CNPG creates -2 then -3
sequentially, each seeded by pg_basebackup from the RUNNING primary. The
primary is not restarted and clients are not disconnected. The anti-affinity
change that DOES cause a switchover is deliberately the next commit.

DEV IMPACT: identical — dev is on the same shape (three single-instance DBs,
all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes
and Longhorn headroom for it (see the next commit's message for the numbers).

Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against
live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning.

Verified: yamllint clean; `kubectl kustomize --load-restrictor
LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0`
both render `instances: 3`.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)

2026-09-07 23:41:31 -05:00

vms

feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

2026-09-04 22:43:54 -05:00

wave-0

feat(cnpg): instances 3 on all three Clusters — a standby to fail over to

2026-09-07 23:41:31 -05:00

wave-1

fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp

2026-09-07 02:06:52 -05:00

wave-2

feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

2026-09-04 22:43:54 -05:00

README.md

refactor(apps): split monolithic apps Kustomization into dependency waves

2026-08-26 12:19:00 -05:00

README.md

apps

App-tier manifests, rendered from catalog/*.yaml ServiceDefinitions by catalog/render.py and split into dependency-ordered waves (wave-<N>/).

Each wave-<N>/ is reconciled by its own Flux Kustomization apps-wave-<N> (clusters/base/apps-wave-<N>.yaml), which dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates (wait: true) ONLY that wave. A crashlooping workload in a later wave can no longer stall an earlier wave or block new revisions.
wave-<N>/kustomization.yaml and clusters/base/apps-wave-<N>.yaml are generated — run python3 catalog/render.py --emit-waves after onboarding a service; do not hand-edit them.
Wave assignment = longest dependency-path depth over the catalog dependencies: graph; a co-located CNPG database <svc>-db is a leaf and always lands strictly below its consumer.

See ../DESIGN-waves.md for the wave table, Flux topology, and the cutover runbook.