jakman85andClaude Opus 5
b7341b329b
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]>
166 lines
9.1 KiB
Markdown
166 lines
9.1 KiB
Markdown
# cert-issuance — A Certificate is not Ready (ACME issuance is failing)
|
|
|
|
**Alert:** `CertificateNotReady` (warning, 1h)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
|
|
**Expression:** `certmanager_certificate_ready_status{condition="False"} == 1`
|
|
|
|
## What this means here
|
|
|
|
cert-manager cannot issue or renew a certificate. The one that matters is `wildcard-home` in the
|
|
`gateway` namespace — secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
|
|
`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (`letsencrypt-prod` on prod). Every HTTPS listener on the
|
|
shared Gateway terminates with it, so issuance failing is a slow-burning cluster-wide outage: it does
|
|
nothing today and takes down every hostname plus in-cluster SSO when the existing certificate expires.
|
|
The 1h `for` reflects that — a failing order is worth investigating today, not paging at 3am.
|
|
|
|
The solver is **DNS-01 against Cloudflare**, which matters because DNS-01 has a failure mode this
|
|
cluster has already hit:
|
|
|
|
**The self-check, not the challenge, is the usual failure.** On 2026-08-16 the wildcard stalled with the
|
|
challenge at `Presented=true` — the Cloudflare TXT record *was* created, so the API token worked — while
|
|
cert-manager's propagation self-check looped **"DNS record not yet propagated"**. The record was
|
|
publicly resolvable at 1.1.1.1 and 8.8.8.8; the NAT'd cluster's own DNS could not see it, because
|
|
split-horizon CoreDNS answers authoritatively for the zone and never forwards. The fix is live in
|
|
`platform/cert-manager/helmrelease.yaml`:
|
|
|
|
```yaml
|
|
dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
|
|
dns01RecursiveNameserversOnly: true
|
|
```
|
|
|
|
The challenge went pending → valid and the certificate Issued. If you find those two lines missing or
|
|
changed, that is your answer.
|
|
|
|
### Verify the alert can even fire (measured 2026-09-13)
|
|
|
|
**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` returned **0 series** on
|
|
prod — there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules
|
|
evaluate against nothing. Certificate health must be read from the CRs, as below. See
|
|
`/runbooks/targets-down` for the full list of inert rule groups.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
|
|
Where this runbook gives a mutating command it is for an operator, and it says so.
|
|
|
|
## First commands
|
|
|
|
Walk the chain top-down — Certificate → CertificateRequest → Order → Challenge. The failure is almost
|
|
always visible at one specific level:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get certificates.cert-manager.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message,NOTAFTER:.status.notAfter'
|
|
```
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get certificaterequests.cert-manager.io -A; echo ==; \
|
|
kubectl --context k3s-prod get orders.acme.cert-manager.io -A; echo ==; \
|
|
kubectl --context k3s-prod get challenges.acme.cert-manager.io -A -o wide'
|
|
```
|
|
|
|
The controller log, which carries the actual ACME error text:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=150 2>&1 | grep -iE "error|propagat|challenge|order|rate" | tail -30'
|
|
```
|
|
|
|
Is the TXT record really there? **Ask the authority, never a resolver** — 1.1.1.1 has served a deleted
|
|
TXT record here for minutes and produced a false failure diagnosis:
|
|
|
|
```
|
|
dig +short @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT
|
|
dig +short @1.1.1.1 _acme-challenge.home.keithjackson.dev TXT
|
|
```
|
|
|
|
Is the resolver override still configured?
|
|
|
|
```
|
|
grep -n "dns01Recursive" platform/cert-manager/helmrelease.yaml
|
|
```
|
|
|
|
The issuer and its credential:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusterissuers.cert-manager.io \
|
|
-o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The DNS-01 propagation self-check cannot resolve the TXT record
|
|
The recorded failure. Signature: Challenge `Presented=true`, status looping on *"DNS record not yet
|
|
propagated"*, and `dig @betty.ns.cloudflare.com` **does** return the TXT. Cause: cert-manager is asking
|
|
the cluster's own resolver, which is split-horizon CoreDNS answering authoritatively for the zone and
|
|
never forwarding. Remediation: the two `dns01Recursive*` values above, in git.
|
|
|
|
### 2. The Cloudflare API token is wrong, expired, or scoped too narrowly
|
|
Signature: Challenge never reaches `Presented=true`, and the log carries a Cloudflare 403/6003. The
|
|
token lives in a SOPS-encrypted Secret. Note the standing rule from the OIDC incident: **non-secret
|
|
environment-specific values must never live inside an encrypted Secret** — `${BASE_DOMAIN}`
|
|
substitution cannot reach inside one, and that is precisely how prod's OIDC clients once ended up
|
|
pointed at dev's Keycloak. Keep hostnames out; keep only the token in.
|
|
|
|
### 3. `EXTERNALDNS_DOMAIN_FILTER` / zone confusion
|
|
Recorded as finding F-DNS1: the filter **must be the ZONE** (`keithjackson.dev`), not the subdomain.
|
|
A zone mismatch means the TXT record is created in the wrong place or not at all.
|
|
|
|
### 4. Two clusters racing the same `_acme-challenge` record
|
|
`infra/PROD-PROVISIONING.md` §10: after the 2026-09-04 incident `platform-config` would have requested a
|
|
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
|
|
clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
|
|
**neither would reliably solve DNS-01**. Only the tier being suspended prevented it. If any second
|
|
cluster exists, establish which one owns the zone first.
|
|
|
|
### 5. Let's Encrypt rate limit
|
|
`too many certificates already issued` in the log. One wildcard is well inside the limits, so this means
|
|
something has been retrying in a loop. Stop the loop and wait out the window.
|
|
|
|
### 6. A new Certificate for a hostname that does not resolve
|
|
A newly added service whose DNS is not yet published cannot solve DNS-01. Check ExternalDNS first.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate if `wildcard-home` has been not-Ready for more than a day**, and check
|
|
`/runbooks/cert-expiry` for how much runway the existing certificate has. Under 3 days of runway is an
|
|
operator page.
|
|
- **Never switch `${ACME_CLUSTER_ISSUER}` to letsencrypt-staging** to get a green certificate. Staging
|
|
issues fine and then breaks every in-cluster OIDC client with `x509: certificate signed by unknown
|
|
authority` — that is the bug that forced LE-prod in the first place.
|
|
- **Do not delete the existing `wildcard-home-tls` Secret** while issuance is broken. A stale but valid
|
|
certificate is far better than none.
|
|
- **Do not create certificates by hand.** cert-manager owns them; the durable change is in git.
|
|
- Rotating the Cloudflare token is an operator action (SOPS, DR path).
|
|
|
|
---
|
|
Source of truth: `runbooks/cert-issuance.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers"
|
|
(commit ~`59dd3d9`) and the LE-staging → LE-prod switch; `infra/PROD-PROVISIONING.md` §10.
|