jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
149 lines
9.1 KiB
Markdown
149 lines
9.1 KiB
Markdown
# cert-expiry — A certificate is expiring soon, or imminently
|
|
|
|
**Alerts:** `CertificateExpiringSoon` (warning, <14d, 1h) and `CertificateExpiryImminent` (critical, <3d, 1h)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.certificates`
|
|
**Expression:** `(certmanager_certificate_expiration_timestamp_seconds - time()) < (14 * 24 * 3600)` / `(3 * 24 * 3600)`
|
|
|
|
## What this means here
|
|
|
|
cert-manager has a Certificate whose expiry is inside 14 days (warning) or 3 days (critical) and it has
|
|
not renewed. cert-manager normally renews at two-thirds of lifetime — about 60 days into a 90-day Let's
|
|
Encrypt certificate — so **by the time this fires, renewal has already been failing for weeks**. Treat
|
|
it as a late symptom of `/runbooks/cert-issuance`, not as a separate problem: the fix is almost always
|
|
"why is issuance failing", and expiry is the clock running out on that.
|
|
|
|
What is at stake concretely. There is effectively **one certificate that matters**: `wildcard-home` in
|
|
the `gateway` namespace, secret `wildcard-home-tls`, requesting **both** `*.${BASE_DOMAIN}` **and**
|
|
`${BASE_DOMAIN}` from `${ACME_CLUSTER_ISSUER}` (= `letsencrypt-prod` on prod). Every HTTPS listener on
|
|
the shared Gateway terminates with it — the wildcard `https` listener and the `https-apex` listener both
|
|
reference `wildcard-home-tls`. If it expires:
|
|
|
|
- every public hostname serves an untrusted certificate;
|
|
- **SSO breaks in-cluster**, because OIDC clients validate the issuer's TLS. This is not hypothetical:
|
|
while the wildcard was still LE-**staging**, in-cluster clients rejected it with
|
|
`x509: certificate signed by unknown authority`, which is what forced the move to `letsencrypt-prod`;
|
|
- and the page you are reading right now is served over that same certificate.
|
|
|
|
**There is no history of an actual expiry on this cluster.** The recorded history is all issuance
|
|
failures. Say so rather than inventing precedent — but note the corollary: the renewal path has never
|
|
been exercised under stress, so do not assume it will quietly work.
|
|
|
|
### Verify the alert can even fire (measured 2026-09-13)
|
|
|
|
**This alert is currently INERT.** `count(certmanager_clock_time_seconds)` and
|
|
`count(certmanager_certificate_expiration_timestamp_seconds)` both returned **0 series** on prod:
|
|
there is no cert-manager scrape job among Prometheus's 19 jobs, so all three certificate rules evaluate
|
|
against nothing. **Certificate expiry is unmonitored today.** Check expiry by hand with the commands
|
|
below, and see `/runbooks/targets-down` for the full list of inert rule groups. Adding cert-manager's
|
|
ServiceMonitor is open work.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Every certificate and its real expiry, straight from the CRs (works with no metrics):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get certificates.cert-manager.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime,ISSUER:.spec.issuerRef.name'
|
|
```
|
|
|
|
What the Gateway is actually serving (the answer that matters), read from outside the cluster with an
|
|
explicit SNI — testing HTTPS from **inside** a pod is misleading here, and an IP+SNI probe without the
|
|
right SNI returns curl exit 35:
|
|
|
|
```
|
|
curl -sv --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/runbooks/ 2>&1 | grep -E "expire|subject|issuer|SSL certificate"
|
|
```
|
|
|
|
```
|
|
echo | openssl s_client -connect 192.168.2.20:443 -servername grafana.home.keithjackson.dev 2>/dev/null | openssl x509 -noout -subject -issuer -dates
|
|
```
|
|
|
|
Is a renewal in flight or stuck?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get certificaterequests.cert-manager.io,orders.acme.cert-manager.io,challenges.acme.cert-manager.io -A; echo; \
|
|
kubectl --context k3s-prod -n cert-manager logs deploy/cert-manager --tail=80 | tail -40'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. Issuance has been failing since the last successful renewal
|
|
The dominant cause, and the one with real history. Go to `/runbooks/cert-issuance` — it covers the
|
|
DNS-01 self-check failure (fixed with `dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` and
|
|
`dns01RecursiveNameserversOnly: true`), the Cloudflare token, and the ACME order/challenge chain. Fixing
|
|
issuance fixes expiry; nothing else does.
|
|
|
|
### 2. Two clusters racing the same `_acme-challenge` TXT record
|
|
A specific, recorded hazard for any future two-cluster window. `infra/PROD-PROVISIONING.md` §10 records
|
|
that after the 2026-09-04 incident, `platform-config` **would have** requested a
|
|
`*.home.keithjackson.dev` wildcard from letsencrypt-prod for a certificate dev already held; both
|
|
clusters would then have raced each other's `_acme-challenge` TXT records on one Cloudflare zone and
|
|
**neither would reliably solve DNS-01**. It never fired only because the tier was suspended. If a second
|
|
cluster exists, confirm which one owns the zone before renewing.
|
|
|
|
### 3. Let's Encrypt rate limits
|
|
One wildcard is well inside the limits, so this is unlikely — unless something has been retrying in a
|
|
loop. `cert-manager` logs will say `too many certificates already issued`. Remediation: stop the loop,
|
|
wait out the window; do **not** switch to letsencrypt-staging as a workaround, because in-cluster OIDC
|
|
clients reject the staging chain (that is the bug that forced LE-prod).
|
|
|
|
### 4. The issuer or its credential changed
|
|
`${ACME_CLUSTER_ISSUER}` is a cluster-vars knob; the Cloudflare API token is a SOPS-encrypted Secret.
|
|
Note the standing rule: **non-secret, environment-specific values must not live inside an encrypted
|
|
Secret** — `${BASE_DOMAIN}` substitution can never reach inside one, which is exactly how prod's OIDC
|
|
clients once ended up pointed at dev's Keycloak.
|
|
|
|
### 5. The Certificate is Ready but the Secret is stale
|
|
Check the Secret's actual notAfter (the `openssl` command above) rather than the CR's status. The Gateway
|
|
reads the Secret.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **`CertificateExpiryImminent` on `wildcard-home` is a cluster-wide outage in under 3 days**, including
|
|
SSO and including this runbook server. Get the operator immediately.
|
|
- **Do not switch to `letsencrypt-staging`** to make issuance succeed. It will issue, and it will break
|
|
every in-cluster OIDC client with `x509: unknown authority`.
|
|
- **Do not delete the `wildcard-home-tls` Secret** to force a reissue while issuance is broken — you
|
|
would replace a certificate that still has days left with none at all.
|
|
- **Do not hand-edit the Secret.** cert-manager owns it; the durable change is in git.
|
|
- Verify DNS at the **authority** (`dig @betty.ns.cloudflare.com _acme-challenge.home.keithjackson.dev TXT`),
|
|
never at a resolver: 1.1.1.1 has served a deleted TXT record for minutes here and produced a false
|
|
failure diagnosis.
|
|
|
|
---
|
|
Source of truth: `runbooks/cert-expiry.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` "Fix 5 — cert-manager DNS-01 self-check via public resolvers" and the
|
|
LE-staging → LE-prod switch (Session 7); `infra/PROD-PROVISIONING.md` §10 (the queued ACME race).
|