home-network-k3s-impl/runbooks/cert-expiry.md

# cert-expiry — A certificate is expiring soon, or imminently

**Alerts:** `CertificateExpiringSoon` (warning, <14d, 1h) and `CertificateExpiryImminent` (critical, <3d, 1h)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
**Expression:** `(certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)` / `(3 * 24 * 3600)`

## What this means here

cert-manager has a Certificate whose expiry is inside 14 days (warning) or 3 days (critical) and it has
not renewed. cert-manager normally renews at two-thirds of lifetime — about 60 days into a 90-day Let's
Encrypt certificate — so **by the time this fires, renewal has already been failing for weeks**. Treat
it as a late symptom of `/runbooks/cert-issuance`, not as a separate problem: the fix is almost always
"why is issuance failing", and expiry is the clock running out on that.

What is at stake concretely. There is effectively **one certificate that matters**: `wildcard-home` in
the `gateway` namespace, secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (= `letsencrypt-prod` on prod). Every HTTPS listener on
the shared Gateway terminates with it — the wildcard `https` listener and the `https-apex` listener both
reference `wildcard-home-tls`. If it expires:

- every public hostname serves an untrusted certificate;
- **SSO breaks in-cluster**, because OIDC clients validate the issuer's TLS. This is not hypothetical:
  while the wildcard was still LE-**staging**, in-cluster clients rejected it with
  `x509: certificate signed by unknown authority`, which is what forced the move to `letsencrypt-prod`;
- and the page you are reading right now is served over that same certificate.

**There is no history of an actual expiry on this cluster.** The recorded history is all issuance
failures. Say so rather than inventing precedent — but note the corollary: the renewal path has never
been exercised under stress, so do not assume it will quietly work.

### Verify the alert can even fire (measured 2026-09-13)

**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` and
`count(certmanager_certificate_expiration_timestamp_seconds)` both returned **0 series** on prod:
there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules evaluate
against nothing. **Certificate expiry is unmonitored today.** Check expiry by hand with the commands
below, and see `/runbooks/targets-down` for the full list of inert rule groups. Adding cert-manager's
ServiceMonitor is open work.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Every certificate and its real expiry, straight from the CRs (works with no metrics):

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get certificates.cert-manager.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime,ISSUER:.spec.issuerRef.name'
```

What the Gateway is actually serving (the answer that matters), read from outside the cluster with an
explicit SNI — testing HTTPS from **inside** a pod is misleading here, and an IP+SNI probe without the
right SNI returns curl exit 35:

```
curl -sv --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/runbooks/ 2>&1 | grep -E "expire|subject|issuer|SSL certificate"
```

```
echo | openssl s_client -connect 192.168.2.20:443 -servername grafana.home.keithjackson.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates
```

Is a renewal in flight or stuck?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get certificaterequests.cert-manager.io,orders.acme.cert-manager.io,challenges.acme.cert-manager.io -A; echo; \
   kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=80 | tail -40'
```

## Likely causes, most likely first

### 1. Issuance has been failing since the last successful renewal
The dominant cause, and the one with real history. Go to `/runbooks/cert-issuance` — it covers the
DNS-01 self-check failure (fixed with `dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` and
`dns01RecursiveNameserversOnly: true`), the Cloudflare token, and the ACME order/challenge chain. Fixing
issuance fixes expiry; nothing else does.

### 2. Two clusters racing the same `_acme-challenge` TXT record
A specific, recorded hazard for any future two-cluster window. `infra/PROD-PROVISIONING.md` §10 records
that after the 2026-09-04 incident, `platform-config` **would have** requested a
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
**neither would reliably solve DNS-01**. It never fired only because the tier was suspended. If a second
cluster exists, confirm which one owns the zone before renewing.

### 3. Let's Encrypt rate limits
One wildcard is well inside the limits, so this is unlikely — unless something has been retrying in a
loop. `cert-manager` logs will say `too many certificates already issued`. Remediation: stop the loop,
wait out the window; do **not** switch to letsencrypt-staging as a workaround, because in-cluster OIDC
clients reject the staging chain (that is the bug that forced LE-prod).

### 4. The issuer or its credential changed
`${ACME_CLUSTER_ISSUER}` is a cluster-vars knob; the Cloudflare API token is a SOPS-encrypted Secret.
Note the standing rule: **non-secret, environment-specific values must not live inside an encrypted
Secret** — `${BASE_DOMAIN}` substitution can never reach inside one, which is exactly how prod's OIDC
clients once ended up pointed at dev's Keycloak.

### 5. The Certificate is Ready but the Secret is stale
Check the Secret's actual notAfter (the `openssl` command above) rather than the CR's status. The Gateway
reads the Secret.

## When to escalate or stop

- **`CertificateExpiryImminent` on `wildcard-home` is a cluster-wide outage in under 3 days**, including
  SSO and including this runbook server. Get the operator immediately.
- **Do not switch to `letsencrypt-staging`** to make issuance succeed. It will issue, and it will break
  every in-cluster OIDC client with `x509: unknown authority`.
- **Do not delete the `wildcard-home-tls` Secret** to force a reissue while issuance is broken — you
  would replace a certificate that still has days left with none at all.
- **Do not hand-edit the Secret.** cert-manager owns it; the durable change is in git.
- Verify DNS at the **authority** (`dig @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT`),
  never at a resolver: 1.1.1.1 has served a deleted TXT record for minutes here and produced a false
  failure diagnosis.

---
Source of truth: `runbooks/cert-expiry.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" and the
LE-staging → LE-prod switch (Session 7); `infra/PROD-PROVISIONING.md` §10 (the queued ACME race).