jakman85andClaude Opus 5.5
18f7ba8d0f
Retire Kasm from the catalog. Drop ACME revocation from R-TLS-02. Make L3 fault injection operator-run and exclude it from the merge gate (R-TEST-03, ADR-0026). Update the gap register to match; Homepage authentication is now ruled must-enforce and stays a gap until the implementation lands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
15 KiB
15 KiB
16 — Known Gaps
Snapshot of implementation main at 50c8719, taken 2026-11-08. This register records where the
implementation does not yet meet a requirement this specification still holds. It exists so that bringing
the specification in line with the implementation did not silently delete the intent the implementation
has yet to reach.
- A gap is not an accepted change. Where the implementation deliberately changed direction, the requirement or an ADR was amended instead, and the item does not appear here.
- This file is a dated snapshot, not a status surface. Every commit here costs the implementation a
ci/SPEC_PINbump, so it is refreshed in batches. Live status comes fromconformance/run.sh --statusin the implementation repo. - Entries were derived by reading both repositories. Nothing here was measured against the running platform.
Operator decisions needed
This cannot be settled by reading either repository.
| Topic | Question | Evidence (implementation repo) |
|---|---|---|
| Bulk-tier photos (R-STATE-03/04) | The bulk tier has no platform backup, and its media dataset includes a photos/ tree. Is that tree irreplaceable, and if so, what protects it? |
docs/BULK-NFS-STORAGE.md §2; platform/velero/volume-policies.yaml |
Environment and provisioning
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|---|---|---|
| R-ENV-01, R-STATE-01 (Phase 1 exit) | A node must be destroyable and rebuildable from code with no data loss; a node was reimaged and rejoined by hand, but no automated destroy/rebuild proof exists on bare metal and the checklist item is unticked. | infra/PROD-PROVISIONING.md sections 0c and 9 |
| docs/02 etcd guardrail (accepted risk) | Shared-device etcd/Longhorn contention is accepted only if etcd write latency stays within budget during a Longhorn rebuild; the measurement failed on node 1 on 2026-09-07 and no dedicated etcd device was added, so the exit condition is unmet. | infra/PROD-PROVISIONING.md section 2 ("The accepted risk, and its exit condition"), section 0c |
| R-PLAT-03, Phase 5 exit | The VM tier must run a working application VM; at the last recorded check (2026-09-14) the UniFi guest had no IP address and its controller ports refused connections. | apps/vms/unifi/README.md |
Storage, state and VMs
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|---|---|---|
| Off-cluster Gitea mirror (R-STATE-03; spec CONTRIBUTING, docs/05 "Circularity & DR") | An off-cluster mirror of the platform repo must outlive the cluster; Gitea has no mirrors and no mirror job exists, while prod's Flux source is in-cluster Gitea. | catalog/gitea.yaml:52; decisions/2026-11-07-gitea-28.md |
| R-STATE-03 (Gitea) | Gitea is DR-critical and its repos and SQLite are backed up by file-level kopia with no quiesce hook, and no Gitea-specific restore has been proven (the DR test restores its own scratch workload). | apps/wave-0/gitea.yaml; conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml |
| R-STATE-03 (UniFi) | Irreplaceable VM state must be recoverable off-cluster; the UniFi disk is excluded on the basis of a configuration export that nothing in the repo declares. | platform/velero/helmrelease.yaml:182-188; apps/vms/unifi/ |
| R-STATE-01 | The per-service rebuild test is an acceptance gate; data-survival and vm-data-survival exist but have no ledger row, and per-service rebuilds are not recorded. |
conformance/runs/ledger.jsonl; conformance/tests/l3-fault-injection/data-survival/ |
| R-STATE-01/03 | End-to-end total-cluster recovery should be drilled at least once; bare-metal rebuild then restore has not been exercised. | conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml (KNOWN, DELIBERATE GAPS) |
| R-STATE-01 (Loki) | Destroying or scaling a workload must leave its PVC; Loki's StatefulSet retention policy deleted its PVC on scale-down (log data is reconstructable). | decisions/2026-10-17-graceful-cluster-shutdown.md |
| R-PLAT-03, R-PLAT-04 | VM create/start/stop/destroy must be proven; vm-lifecycle is logged green only on the decommissioned dev cluster and has no ledger row. |
DECISIONS-LOG.md:896; conformance/runs/ledger.jsonl |
| R-NET-06 (VMs) | An egress: vpn VM must egress through the VPN with a working kill-switch; no VM VPN conformance test exists and vm-demo calls it a follow-up. |
catalog/vm-demo.yaml; conformance/tests/l2-behavioral/vm-lifecycle/ |
| R-UPD-* (UniFi guest) | Guest OS packages must auto-update on schedule with per-VM exceptions recorded; UniFi's cloud-init sets package_upgrade: false and installs no unattended-upgrades. |
apps/vms/unifi/30-virtualmachine.yaml:206-207 |
| R-OBS-* (VM guests) | A VM must expose node-exporter-class metrics from inside the guest; neither the renderer nor UniFi provides any, only KubeVirt/virt platform metrics. | catalog/render.py; platform/kubevirt/servicemonitor.yaml |
| R-STATE-03 (VM disks) | VM disks are meant to share the platform backup story; the backup path is Velero file-system backup, which upstream documents as not supporting block-mode volumes, and the default VM class is block-mode. Unmeasured here: whether any block-mode VM disk can be captured by this path. | platform/velero/helmrelease.yaml:175; platform/longhorn/storageclasses.yaml (longhorn-vm) |
Networking, DNS and TLS
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|---|---|---|
| VPN port forwarding (docs/03, ADR-0015) | The spec requires a host-side PIA port-forward helper for qBittorrent and an alert on PF acquisition failure; neither exists. | infra/ansible/roles/vpn_egress_gateway/README.md:170, platform/vpn-egress/README.md |
| R-NET-01 | Every namespace must carry a default-deny baseline; Kyverno generates it only for app-namespace=true, and no policy file exists for cert-manager, cnpg, external-dns, longhorn, velero, backup, image-automation, external-snapshotter or vpn-egress. |
platform/kyverno/policies/require-default-deny-baseline.yaml, decisions/2026-10-03-public-dns-wan-target.md |
| R-SEC-03, R-OBS-04 | docs/03 requires alerting on unexpected denied flows; HubbleDeniedFlowSpike cannot fire because no Cilium or Hubble metrics are scraped. |
platform/monitoring/alerts.yaml:1047-1050,1162-1173 |
| Client IP (ADR-0016) | IP-based logging and policy must read CF-Connecting-IP / X-Forwarded-For and trust them only from cloudflared; no trusted-proxy or client-IP configuration exists in catalog/, platform/ or apps/. |
platform/cloudflared/config.yaml (no client-IP handling downstream of it) |
| R-DNS-04 | The edge must not forward 80/443; closing them is a manual router step, last recorded as not done, so the state is unverified. | decisions/2026-11-08-dns-via-cloudflare-tunnel.md, runbooks/cloudflare-tunnel.md:47-55 |
| R-DNS-04, R-DNS-05, R-SEC-04 | docs/09 and docs/14 require behavioural proof at the tunnel and from the LAN through a public resolver; only L0 static checks exist, and no L2 test touches the tunnel. | conformance/tests/l0-static/cloudflared-ingress-rules, conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml |
Security, identity, observability and the command center
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|---|---|---|
| R-IDP-02, R-IDP-06 (operator ruling 2026-11-08: enforce it) | No management UI may be exposed unauthenticated and the C2 requires an authorized group, but Homepage at the apex is routed straight to its Service with no gate and its forward-auth annotation enforces nothing. |
platform/homepage/httproute.yaml, platform/homepage/README.md |
| R-SEC-03, R-C2-02 | Operational actions must be audited, but the k3s API server has only OIDC arguments and no audit log or audit policy. | infra/ansible/roles/k3s_server/tasks/main.yml |
| R-SEC-01 | The acceptance gate requires no plaintext secret anywhere in git history (scanned), but the check inspects the tracked tree only and no history scanner runs in CI. | conformance/tests/l0-static/no-plaintext-secrets/check.sh, ci/ |
| R-SEC-01 | Rotation is meant to trigger a workload restart through reconcile (e.g. a checksum annotation), but no generated mechanism exists; the only annotations are hand-stamped on Homepage and Grafana. | platform/homepage/deployment.yaml, platform/monitoring/kube-prometheus-stack.yaml, catalog/render.py |
| R-SEC-01 | VPN keys, DNS tokens and SMTP credentials should have a documented rotation procedure, but the only trace is a one-line note on the Cloudflare token and none exists for VPN keys or SMTP. | runbooks/cert-issuance.md, runbooks/ |
| R-OBS-01 | Alloy is specified to ship node, pod and VM logs, but it collects pod logs only, with no node or journal source. | platform/monitoring/alloy.yaml |
| R-OBS-04 | NetworkPolicy reconcile errors are a specified platform alert, but no such rule exists. | platform/monitoring/alerts.yaml |
| R-OBS-04 | Restore-test failures are a specified platform alert, but only backup and schedule alerts exist. | platform/monitoring/alerts.yaml |
| R-OBS-04 | Keycloak login-endpoint failures are a specified alert, but the repo has no endpoint probe, only workload and database health rules. | platform/monitoring/alerts.yaml |
| R-OBS-01 | Every forward-auth gate is meant to be observed, but Crafty's gate has no metrics listener, so OAuth2ProxyGateUnobserved fires for it by design. |
platform/crafty/deployment.yaml, platform/oauth2-proxy/README.md |
| R-OBS-03 | Per-service dashboards are specified as onboarding output, but the renderer emits no dashboards; the in-repo dashboards are node pressure, KubeVirt and Keycloak only. | catalog/render.py, platform/monitoring/node-pressure-dashboard.yaml, platform/kubevirt/dashboard.yaml, platform/keycloak/grafana-dashboard.yaml |
| R-OBS-01, R-ENV-02 | Log retention is specified as an overlay setting, but Loki's retention is hardcoded with no overlay knob. | platform/monitoring/loki.yaml, platform/monitoring/README.md |
| R-OBS-04 | Monitoring storage is specified on Longhorn, but Alertmanager and Grafana run without persistent volumes, so silences and notification state are lost on restart. | platform/monitoring/kube-prometheus-stack.yaml |
| R-OBS-02, R-OBS-04 | Acceptance requires platform alerts to fire and email in injected-fault tests, but the L3 alerting test is a scaffold. | conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml, CLAUDE.md |
| R-C2-01 | Homepage is specified as a live catalog with status pings and per-service widgets, but it is a hand-maintained ConfigMap of link tiles with no status or widgets. | platform/homepage/configmap.yaml, platform/homepage/README.md |
| R-C2-02 | The C2 must surface platform health and firing alerts, but none is shown on the C2 landing surface; health is reachable only by linking out to Grafana. | platform/homepage/configmap.yaml |
| R-C2-02 | Grafana dashboards are specified as embedded in the C2, but Homepage only links to Grafana. | platform/homepage/configmap.yaml |
| R-C2-02 | Flux status is specified via Headlamp's Flux view, the Flux UI or a Homepage widget, but Headlamp has no Flux plugin configured and Homepage has no widget. | platform/headlamp/helmrelease.yaml, platform/homepage/configmap.yaml |
| R-C2-02 | C2 actions are specified to include VM start/stop, Flux reconcile and alert acknowledgement, but only native Headlamp workload actions exist, with no plugins for the others. | platform/headlamp/helmrelease.yaml |
Service catalog
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|---|---|---|
R-STATE-03, docs/13 backup.policy |
backup.policy must reference a named off-cluster schedule, and daily-offcluster resolves to no schedule object; it is an annotation only, while coverage comes from cluster-wide Velero schedules. |
platform/velero/README.md:33-34, schemas/renderer-contract.md:428, catalog/*.yaml (backup.policy) |
R-UPD-01/02, docs/13 update.cadence |
update.cadence should set how often a service checks or applies updates, and the renderer does not read it for containers or VMs. |
catalog/render.py (no cadence reference), schemas/renderer-contract.md:388-390 |
Delivery, updates and conformance
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|---|---|---|
| R-TEST-03 | Merges require the tests for touched requirements to pass; the gate runs L0/L1 and tagged-coverage only, because L2 does not run in CI (job disabled, trigger path unimplemented, cluster not declared), and a PASS names that declared gap. | .gitea/workflows/gate.yml (conformance job if: false), ci/conformance.sh, docs/plans/l2-l3-conformance-ci-wiring.md |
| R-TEST-03 | Phase exits require the full suite and regression runs continuously; no pipeline runs the full-suite gate and no scheduled run exists, so L2 evidence comes only from operator-recorded runs, last recorded 2026-09-19. L3 is operator-run by design (ADR-0026). | conformance/run.sh (--require-all, --record), conformance/runs/ledger.jsonl |
| R-TEST-01, R-TEST-04 | A requirement with no passing test is not satisfied; all 51 registry requirements have a tagged test but PROVEN is 0 (3 stale: R-NET-06, R-STATE-02, R-STATE-03; 48 not derivable from the ledger). | conformance/lib/status.py, conformance/runs/ledger.jsonl |
| R-OBS-02, R-GITOPS-02, R-GITOPS-03 | Each requirement needs an executable behavior test; the only verifying tests (alerting, onboarding-completeness, gitops-drift) are scaffolds. |
conformance/lib/status.py, conformance/tests/ |
| R-TEST-06 | The spec must be verifiably immutable to agents; CI never checks the spec repo out, so spec-immutability always skips there (tolerated) and only local gates with a sibling checkout enforce it. | .gitea/workflows/gate.yml:193,227, ci/spec-immutability.sh, ci/gate.sh (REQUIRE_SPEC_VERIFICATION) |
| R-UPD-01, R-UPD-02 | Hands-off updates must be proven by a behavior test; the only verifying test exercises Flux image-policy resolution, not the Renovate path that delivers updates. | conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml |
| R-UPD-03 | A pinned service must not advance; the ServiceDefinition update.pin only affects a disabled render path, so it would not stop Renovate (no catalog entry uses it; a Renovate rule is the working pin). |
catalog/render.py:1731, ci/render-golden.sh, renovate.json5 |
| R-UPD-03 | A failed update must auto-roll-back; only HelmReleases have remediation, while plain Deployments stall on readiness and autoRollbackOnFailedHealth renders only onto the disabled image policy. |
catalog/render.py:1747, conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml |
| R-UPD-02, R-OBS-04 | Update failures must alert; the update alerts watch Flux objects that do not exist and so cannot fire, and nothing alerts when Renovate's schedule stops. | platform/monitoring/alerts.yaml:1538,1552, .gitea/workflows/renovate.yaml header |