This website requires JavaScript.
6e4d311545
docs(prod): will 6.12.y get the memattr fix? Researched — not on its own
jakman85
2026-09-01 23:04:56 -05:00
4809199da6
Merge pull request 'Node 2 memory: solved — Linux 6.12 EFI memattr bug, not hardware' (#15 ) from docs/node2-memory-refine into main
gitea-admin
2026-09-02 03:45:30 +00:00
0a733294ff
docs(prod): node 1's DIMMs are not a matched pair
jakman85
2026-09-01 22:31:26 -05:00
cf33338448
feat(node_health): detect the EFI memattr bug on every run, two independent ways
jakman85 and Claude Opus 5
2026-09-01 22:30:45 -05:00
47f02c6e06
docs(prod): second independent confirmation + no cmdline workaround exists
jakman85
2026-09-01 22:16:22 -05:00
49d82491dd
docs(prod): node 2 memory SOLVED — Linux 6.12 EFI memattr bug, NOT a hardware fault
jakman85
2026-09-01 22:14:14 -05:00
5ea9c93f58
docs(prod): BIOS password answer + CORRECT the 0x4 'auth failure' reading
jakman85
2026-09-01 22:10:07 -05:00
3a88abb958
docs(prod): iGPU ruled out as the cause; CORRECT my last_pfn misreading
jakman85
2026-09-01 21:54:12 -05:00
086e9091c9
Merge pull request 'feat(prod): node 2 online — open hardware fault (2.6 GiB of 16 GB) + BIOS drift' (#13 ) from feat/prod-node-2 into main
gitea-admin
2026-09-02 02:49:10 +00:00
9f7052ab2b
docs(prod): fill in node-2 inventory; drop stale investigate bullets
jakman85
2026-09-01 21:45:37 -05:00
3c8f809aaf
docs(prod): CORRECTION — BIOS attributes are read-only in practice, not writable
jakman85
2026-09-01 21:45:15 -05:00
11b21f7347
docs(prod): BIOS settings compared programmatically via hp-bioscfg
jakman85
2026-09-01 21:04:12 -05:00
cb69a53825
feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB
jakman85 and Claude Opus 5
2026-09-01 20:56:58 -05:00
f1daa8cae0
Merge pull request 'PROVEN: 1 → 2 — R-STATE-03 DR restore executed and proven' (#12 ) from feat/r-state-03-proven into main
gitea-admin
2026-09-02 00:24:57 +00:00
380ea98a38
feat(conformance): R-STATE-03 PROVEN — DR restore from off-cluster backup, executed
jakman85 and Claude Opus 5
2026-09-01 19:23:09 -05:00
acae2ad36c
Merge pull request 'PROVEN: 0 → 1 — R-STATE-02 proven, plus two ledger bugs that made proof impossible' (#11 ) from feat/ledger-section-parsing into main
gitea-admin
2026-09-02 00:10:06 +00:00
fdc3265585
fix(conformance): the ledger counted as evidence FOR the requirements it proves
jakman85 and Claude Opus 5
2026-09-01 19:08:38 -05:00
4cb5e7a8d4
feat(conformance): first PROVEN row — R-STATE-02 node-loss
jakman85
2026-09-01 19:07:39 -05:00
57d4df8c4f
fix(conformance): ledger read tokens from Chainsaw's COMMAND echo, not real output
jakman85 and Claude Opus 5
2026-09-01 19:07:26 -05:00
14494e1947
Merge pull request 'feat(conformance): committed run ledger — prerequisite for moving PROVEN off zero' (#10 ) from feat/conformance-run-ledger into main
gitea-admin
2026-09-01 23:50:28 +00:00
e9a4b669d4
feat(conformance): committed run ledger — the only thing that can move PROVEN off zero
jakman85 and Claude Opus 5
2026-09-01 18:49:01 -05:00
aa0a50a143
Merge pull request 'chore(flux): restore platform prune:true — Cilium split merge 2 of 2' (#9 ) from chore/cilium-split-merge2 into main
gitea-admin
2026-09-01 23:41:51 +00:00
22174186ff
chore(flux): restore platform prune:true — MERGE 2 of 2 (Cilium split cleanup)
jakman85 and Claude Opus 5
2026-09-01 18:39:20 -05:00
23aa2854f1
Merge pull request 'feat(flux): Cilium gets its own Kustomization — merge 1 of 2' (#8 ) from feat/cilium-own-kustomization into main
gitea-admin
2026-09-01 23:35:04 +00:00
63f98aab26
feat(flux): give Cilium its own Kustomization — MERGE 1 of 2 (transition)
jakman85 and Claude Opus 5
2026-09-01 18:33:20 -05:00
62968c9ee9
Merge pull request 'docs: P0/P1/P2 landed + postStart crashloop incident' (#7 ) from docs/poststart-incident into main
gitea-admin
2026-09-01 22:02:56 +00:00
645994e3a3
docs: record P0/P1/P2 landing + the postStart crashloop incident
jakman85
2026-09-01 16:59:41 -05:00
ac9f20034c
Merge pull request 'fix(sso): postStart hooks must exit 0 — nextcloud crashloop' (#6 ) from fix/poststart-exit-status into main
gitea-admin
2026-09-01 21:55:38 +00:00
4c4f5355d9
fix(sso): postStart hooks must exit 0 — nextcloud crashlooped the moment OIDC ran
jakman85 and Claude Opus 5
2026-09-01 16:53:08 -05:00
63002d19c8
Merge pull request 'Close the post-build substitution race: dependsOn + strict mode + CNI blast radius' (#5 ) from fix/strict-postbuild-substitutions into main
gitea-admin
2026-09-01 21:30:53 +00:00
6b4da27f84
feat(prod): SOPS decryption (same age key as dev) + GitOps source on main
jakman85 and Claude Opus 5
2026-09-01 16:27:32 -05:00
666a60776a
fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1)
jakman85 and Claude Opus 5
2026-09-01 01:45:04 -05:00
fdc4a7d9f4
fix(cilium): maxUnavailable 2 -> 1 — bound CNI rollout blast radius (P2)
jakman85 and Claude Opus 5
2026-09-01 01:40:20 -05:00
a344b046c9
docs: research findings — measured Flux substitution semantics + ranked remediation plan
jakman85
2026-09-01 01:33:16 -05:00
8a2be824f6
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
jakman85 and Claude Opus 5
2026-09-01 01:31:07 -05:00
3484955903
fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard
jakman85 and Claude Opus 5
2026-09-01 01:29:40 -05:00
b8602c939c
docs: INCIDENT — Flux post-build substitution fails OPEN, briefly broke Cilium
jakman85
2026-09-01 01:00:16 -05:00
dce0e8ffe9
Merge pull request 'Phase 7 (prod) opened — prep only: T-7.1 checklist, R-ENV-02 tokenization, node_health, CI false-green fix' (#4 ) from docs/phase-7-open-prod-node-1 into main
gitea-admin
2026-09-01 05:43:25 +00:00
ff0533106b
feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path
jakman85 and Claude Opus 5
2026-09-01 00:40:29 -05:00
0b3db6e778
fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals
jakman85 and Claude Opus 5
2026-09-01 00:40:29 -05:00
ce1966acb0
docs: PM handoff — Phase-7 prep session complete (PR #4 , 13 commits, CI green)
jakman85 and Claude Opus 5
2026-09-01 00:15:28 -05:00
c1f24e4a6d
fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
jakman85 and Claude Opus 5
2026-09-01 00:11:49 -05:00
14be9b68f6
test(conformance): dr-restore actually asserts off-cluster restore (R-STATE-03)
jakman85 and Claude Opus 5
2026-08-31 23:53:36 -05:00
e623d27e2d
feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)
jakman85 and Claude Opus 5
2026-08-31 23:53:36 -05:00
1157c1b071
fix(gitea-actions): wait for dind to ACCEPT connections, not just write certs
jakman85 and Claude Opus 5
2026-08-31 23:49:43 -05:00
6e77150ae1
fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
jakman85 and Claude Opus 5
2026-08-31 23:26:49 -05:00
9f59c36000
feat(ansible): node_health role — read-only hardware health for prod + dev nodes
jakman85 and Claude Opus 5
2026-08-31 23:20:25 -05:00
677f9ea5f0
docs: CI is the enforcement point; local gate produced a FALSE GREEN
jakman85 and Claude Opus 5
2026-08-31 23:05:57 -05:00
a1fb31a7ef
docs(prod): record the decided prod address plan; packages-in-role correction
jakman85 and Claude Opus 5
2026-08-31 22:44:10 -05:00
1252823d80
feat(conformance): repo-derived spec status report (conformance/run.sh --status)
jakman85 and Claude Opus 5
2026-08-31 22:44:10 -05:00
666cc0d809
feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline
jakman85 and Claude Opus 5
2026-08-31 22:44:10 -05:00
3f980d2a17
feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
jakman85 and Claude Opus 5
2026-08-31 22:43:36 -05:00
e3c44368c8
docs(prod): correct sudo posture to vaulted become; record status-surface ruling
jakman85 and Claude Opus 5
2026-08-31 22:29:39 -05:00
574088dfeb
docs(prod): open Phase 7 (prep only) + T-7.1 bare-metal provisioning checklist
jakman85 and Claude Opus 5
2026-08-31 22:17:51 -05:00
c182888133
docs: PM handoff — 2026-08-31 current-state, gotchas + next steps; PHASE1-STATUS 8 vCPU
jakman85 and Claude Opus 4.8
2026-08-31 21:23:04 -05:00
7a79cea94a
fix(flux): vendor external-snapshotter manifests locally (stop remote-base build flake)
jakman85 and Claude Opus 4.8
2026-08-31 10:53:09 -05:00
d1a547fd15
docs(plan): L2/L3 conformance CI wiring plan; gate.yml placeholder -> plan pointer
jakman85 and Claude Opus 4.8
2026-08-31 09:01:18 -05:00
27bc02471c
ci: exclude comment-only prose from touched-requirements scan; gate.yml -> make -C ci gate
jakman85 and Claude Opus 4.8
2026-08-31 08:32:39 -05:00
b2752aec48
feat(auth): gate jackett/qbittorrent/radarr/sonarr UIs with oauth2-proxy sidecar (rollout)
jakman85 and Claude Opus 4.8
2026-08-31 08:45:46 -05:00
72d4720d58
feat(auth): gate pihole UI with per-app oauth2-proxy sidecar (canary)
jakman85 and Claude Opus 4.8
2026-08-31 08:36:00 -05:00
a04251bbc8
docs: plan for Longhorn SSD tiering (hot-tier move off HDD)
jakman85 and Claude Opus 4.8
2026-08-31 08:27:21 -05:00
d60e5531b1
test(conformance): correct stale ImageVolume-gate rationale for k3s v1.36.4
jakman85 and Claude Opus 4.8
2026-08-31 01:19:36 -05:00
b745bab26b
docs(plan): evaluate Cilium 1.20.1 native Gateway ExternalAuth vs oauth2-proxy
jakman85 and Claude Opus 4.8
2026-08-31 01:19:36 -05:00
676c37a247
tofu: node vCPU 6 -> 8 (CPU spike headroom)
jakman85 and Claude Opus 4.8
2026-08-30 23:47:49 -05:00
8a424bcba3
test(l1): regenerate render goldens for the keycloak-egress renderer change
jakman85 and Claude Opus 4.8
2026-08-30 23:35:20 -05:00
68690e5019
fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
jakman85 and Claude Opus 4.8
2026-08-30 23:34:03 -05:00
45deac9d50
keycloak: fix conditional-OTP role config key (blocked all admin SSO with 403)
jakman85 and Claude Opus 4.8
2026-08-30 21:56:13 -05:00
a720c73fcd
monitoring: node CPU-pressure alerts + PSI/load dashboard (#4 )
jakman85 and Claude Opus 4.8
2026-08-30 21:51:15 -05:00
3e61f817b6
kps: raise Prometheus (2->3Gi) + Grafana (512->768Mi) memory limits
jakman85 and Claude Opus 4.8
2026-08-30 21:33:40 -05:00
1f921fbfed
docs(ansible): add re-provision runbook + node-host playbook list
jakman85 and Claude Opus 4.8
2026-08-30 21:18:29 -05:00
22db5fdb63
kyverno: 3.3/v1.13 -> 3.9/v1.19 (stay-current) + CLI lockstep
jakman85 and Claude Opus 4.8
2026-08-30 16:28:16 -05:00
954b9616b9
loki: chart 6.x -> 7.x (Loki 3.6.x; stay-current)
jakman85 and Claude Opus 4.8
2026-08-30 16:21:32 -05:00
cbc2301b30
docs: reconcile docs/comments with post-upgrade cluster state
jakman85 and Claude Opus 4.8
2026-08-30 15:13:12 -05:00
dbd73f1f88
kube-vip: v0.8.9 -> v1.2.3 (k8s 1.36 catch-up)
jakman85 and Claude Opus 4.8
2026-08-30 14:58:33 -05:00
b2b1aa55e2
kps: bump Grafana memory limit 256Mi->512Mi (Grafana 12.x OOMKill was root cause of 65->88 upgrade timeout)
jakman85 and Claude Opus 4.8
2026-08-30 14:52:06 -05:00
00d895d25f
kps: 20m timeout + no-rollback remediation for the 65->88 catch-up
jakman85 and Claude Opus 4.8
2026-08-30 12:08:29 -05:00
bbe08e6762
velero: chart 8.x->12.x (Velero 1.15->1.18), plugin-for-aws v1.14.2 (k8s 1.36 catch-up)
jakman85 and Claude Opus 4.8
2026-08-30 11:55:29 -05:00
10b5228105
cert-manager sequential upgrade -> 1.21 (k8s 1.36 catch-up, final hop)
jakman85 and Claude Opus 4.8
2026-08-30 11:52:42 -05:00
c868382f5a
cert-manager sequential upgrade -> 1.20 (k8s 1.36 catch-up)
jakman85 and Claude Opus 4.8
2026-08-30 11:48:26 -05:00
e67eee527a
cert-manager sequential upgrade -> 1.19 (k8s 1.36 catch-up)
jakman85 and Claude Opus 4.8
2026-08-30 11:42:40 -05:00
e42c4a32f1
cert-manager sequential upgrade -> 1.18 (k8s 1.36 catch-up)
jakman85 and Claude Opus 4.8
2026-08-30 11:35:18 -05:00
7d2f92b25e
cert-manager -> 1.17 (k8s 1.36 catch-up, hop 1/5)
jakman85 and Claude Opus 4.8
2026-08-30 11:26:05 -05:00
037aa7fe9a
Component catch-up: kube-prometheus-stack 65->88 (po v0.93, k8s 1.36); CRDs pre-applied
jakman85 and Claude Opus 4.8
2026-08-30 11:23:13 -05:00
19a368b237
Component catch-up batch 1: external-dns 1.21, oauth2-proxy 10.x, Alloy 1.x, CNPG in-place flag
jakman85 and Claude Opus 4.8
2026-08-30 11:21:55 -05:00
ba72725a4e
OS upgrade complete: Debian 12->13 (trixie) baseline + re-provision role fixes
jakman85 and Claude Opus 4.8
2026-08-30 11:12:53 -05:00
d84feac736
OS upgrade complete: Debian 12->13 (trixie) baseline + re-provision role fixes
jakman85 and Claude Opus 4.8
2026-08-30 11:09:57 -05:00
ef0a735145
node_baseline: add cryptsetup (Longhorn 1.12 RequiredPackages) for Debian 13 re-provision
jakman85 and Claude Opus 4.8
2026-08-30 10:12:53 -05:00
7a59ef07bb
Upgrade complete: pin k3s_version v1.31.5 -> v1.36.4+k3s1
jakman85 and Claude Opus 4.8
2026-08-30 02:17:17 -05:00
6bc4406dca
W5 hardening: Longhorn HR timeout 15m + no downgrade-rollback
jakman85 and Claude Opus 4.8
2026-08-30 01:04:15 -05:00
b4b3bb0922
W5 (FINAL): Cilium 1.19.7->1.20.1 (+GW-API CRDs v1.6.1), Longhorn 1.11->1.12.1 (pre-k8s-1.36)
jakman85 and Claude Opus 4.8
2026-08-30 00:22:53 -05:00
39cc09e6b3
W4: Longhorn 1.10->1.11.3 (pre-k8s-1.35; Cilium/GW-API CRDs held steady)
jakman85 and Claude Opus 4.8
2026-08-30 00:13:11 -05:00
36173b9529
W3: Cilium 1.18.13->1.19.7 (+GW-API CRDs v1.4.1), Longhorn 1.9->1.10.2 (pre-k8s-1.34)
jakman85 and Claude Opus 4.8
2026-08-30 00:00:23 -05:00
cd8cf6220b
W2: Cilium 1.17.18->1.18.13 (+GW-API CRDs v1.3.0), Longhorn 1.8->1.9 (pre-k8s-1.33)
jakman85 and Claude Opus 4.8
2026-08-29 23:46:09 -05:00
5efd983239
W1: Cilium 1.16.5->1.17.18, Longhorn 1.7->1.8 (pre-k8s-1.32 component landing)
jakman85 and Claude Opus 4.8
2026-08-29 23:23:34 -05:00
dd8238a017
Migrate Flux image-automation API v1beta2 -> v1 (pre-Flux-2.7 upgrade)
jakman85 and Claude Opus 4.8
2026-08-29 23:22:35 -05:00
d24a443c7d
chore(flux): migrate image-automation API v1beta2 -> v1
jakman85 and Claude Opus 4.8
2026-08-29 23:19:20 -05:00
242951a000
fix(gitea-actions): wait for dind daemon to accept connections before starting runner
jakman85 and Claude Opus 4.8
2026-08-29 22:28:20 -05:00
8dfb42f6c0
docs: add Phase 2 SSD-etcd execution runbook
jakman85 and Claude Opus 4.8
2026-08-29 21:28:46 -05:00
e1556e3004
Merge SSD-backed etcd isolation: virtio2 disk per node + node_baseline mount
jakman85 and Claude Opus 4.8
2026-08-29 21:28:41 -05:00
af5fc5b25f
feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan)
jakman85 and Claude Opus 4.8
2026-08-29 17:21:06 -05:00
jakman85
gitea-admin
