Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8427b28b0e | ||
|
|
64c72fd394 | ||
|
|
1ea73544c9 | ||
|
|
296d6d3275 | ||
|
|
0c6949ed98 |
No files matched your search
@@ -131,6 +131,10 @@ jobs:
|
|||||||
# merging. Green here means the token resolved and the live Gitea matched the
|
# merging. Green here means the token resolved and the live Gitea matched the
|
||||||
# declaration in infra/gitea/expected-branch-protection.json.
|
# declaration in infra/gitea/expected-branch-protection.json.
|
||||||
REQUIRE_GITEA_SETTINGS: "true"
|
REQUIRE_GITEA_SETTINGS: "true"
|
||||||
|
# ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
|
||||||
|
# PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
|
||||||
|
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||||
|
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||||
|
|
||||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||||
|
|||||||
@@ -14,7 +14,28 @@ There is no other path. Specifically:
|
|||||||
|
|
||||||
- **Do not `git push origin main`.**
|
- **Do not `git push origin main`.**
|
||||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||||
- **Do not merge your own PR.** A human approves and clicks merge.
|
- **Do not merge your own PR.** See the approval model below — and either way, the merge is
|
||||||
|
someone else's action, never yours.
|
||||||
|
|
||||||
|
### Who approves
|
||||||
|
|
||||||
|
**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
|
||||||
|
branch requires no human approval for one. That is a deliberate transfer of authority, on the
|
||||||
|
operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
|
||||||
|
good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
|
||||||
|
add a human.
|
||||||
|
|
||||||
|
**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
|
||||||
|
admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
|
||||||
|
approving review from someone who is not its author. The reason is narrow and structural — **a
|
||||||
|
change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
|
||||||
|
the weakened check. Everything else merges on green.
|
||||||
|
|
||||||
|
The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
|
||||||
|
sensitive path, so the policy cannot be widened or narrowed without a human.
|
||||||
|
|
||||||
|
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
|
||||||
|
missing credential is a lock-out, here a skip would be a bypass.
|
||||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||||
|
|
||||||
|
|||||||
@@ -35,6 +35,9 @@ decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-
|
|||||||
l1: ## L1 rendering checks (no cluster; live now)
|
l1: ## L1 rendering checks (no cluster; live now)
|
||||||
@"$(CI_DIR)l1.sh"
|
@"$(CI_DIR)l1.sh"
|
||||||
|
|
||||||
|
approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
|
||||||
|
@./approval-policy.sh
|
||||||
|
|
||||||
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
|
||||||
@./gitea-settings.sh
|
@./gitea-settings.sh
|
||||||
|
|
||||||
|
|||||||
Executable
+117
@@ -0,0 +1,117 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
|
||||||
|
# what the pipeline verifies or the security posture it rests on.
|
||||||
|
#
|
||||||
|
# THE POLICY, and why it is shaped this way.
|
||||||
|
#
|
||||||
|
# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
|
||||||
|
# pipeline is not good enough to be trusted for this, then it is not good enough to do its
|
||||||
|
# intended job and needs to be improved. Manual approvers should only be for architecturally or
|
||||||
|
# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
|
||||||
|
#
|
||||||
|
# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
|
||||||
|
# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
|
||||||
|
# A PR that weakens a check would be approved by the weakened check. The same applies to the
|
||||||
|
# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
|
||||||
|
# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
|
||||||
|
# requires a human for them.
|
||||||
|
#
|
||||||
|
# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
|
||||||
|
# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
|
||||||
|
# policy cannot be widened without a human.
|
||||||
|
#
|
||||||
|
# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
|
||||||
|
# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
|
||||||
|
# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
|
||||||
|
# BYPASS, not a gap. So: cannot determine -> BLOCK.
|
||||||
|
#
|
||||||
|
# EXIT CODES
|
||||||
|
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
|
||||||
|
# non-author APPROVED review is present
|
||||||
|
# 1 sensitive paths touched without such a review, OR the policy could not be evaluated
|
||||||
|
set -euo pipefail
|
||||||
|
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source "${CI_DIR}/lib.sh"
|
||||||
|
|
||||||
|
log "Stage: approval policy (CI approves; humans approve changes to CI)"
|
||||||
|
|
||||||
|
DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
|
||||||
|
[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
|
||||||
|
require_tool jq
|
||||||
|
require_tool python3
|
||||||
|
|
||||||
|
# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
|
||||||
|
# by branch protection, so this is genuinely not-applicable rather than a skipped check.
|
||||||
|
EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
|
||||||
|
PR="${PR_NUMBER:-}"
|
||||||
|
if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
|
||||||
|
ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
|
||||||
|
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
|
||||||
|
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
|
||||||
|
if [ -z "$TOKEN" ]; then
|
||||||
|
err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
|
||||||
|
err " BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
|
||||||
|
summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||||
|
|
||||||
|
FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
|
||||||
|
PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
|
||||||
|
REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
|
||||||
|
for v in FILES_JSON PR_JSON REVIEWS_JSON; do
|
||||||
|
jq -e . >/dev/null 2>&1 <<<"${!v}" || {
|
||||||
|
err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
|
||||||
|
summary "- approval-policy: **BLOCKED — Gitea unreadable**"
|
||||||
|
exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
|
||||||
|
[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
|
||||||
|
|
||||||
|
# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
|
||||||
|
# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
|
||||||
|
# never read. Caught before shipping, by running it.
|
||||||
|
SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
|
||||||
|
import json, os, fnmatch
|
||||||
|
decl = json.load(open(os.environ["DECL"]))
|
||||||
|
files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
|
||||||
|
pats = decl["sensitive_paths"]
|
||||||
|
hits = sorted({f for f in files for p in pats
|
||||||
|
if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
|
||||||
|
print("\n".join(hits))
|
||||||
|
')"
|
||||||
|
|
||||||
|
if [ -z "$SENSITIVE" ]; then
|
||||||
|
ok "no architecturally/security-relevant paths touched — the GATE is the approver."
|
||||||
|
summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo " sensitive paths touched:" >&2
|
||||||
|
printf ' %s\n' $SENSITIVE >&2
|
||||||
|
|
||||||
|
APPROVER="$(jq -r --arg a "$AUTHOR" \
|
||||||
|
'[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
|
||||||
|
<<<"$REVIEWS_JSON")"
|
||||||
|
|
||||||
|
if [ -n "$APPROVER" ]; then
|
||||||
|
ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
|
||||||
|
summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
|
||||||
|
err " review from anyone other than its author ('${AUTHOR}')."
|
||||||
|
err " A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
|
||||||
|
err " honestly approve for itself. Get a human review, or move the change out of:"
|
||||||
|
printf ' %s\n' $SENSITIVE >&2
|
||||||
|
err " The list lives in infra/gitea/approval-policy.json, with a reason per entry."
|
||||||
|
summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
|
||||||
|
exit 1
|
||||||
+15
@@ -113,6 +113,7 @@ run_stage lint "${CI_DIR}/lint.sh"
|
|||||||
run_stage l0 "${CI_DIR}/l0.sh"
|
run_stage l0 "${CI_DIR}/l0.sh"
|
||||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||||
run_stage l1 "${CI_DIR}/l1.sh"
|
run_stage l1 "${CI_DIR}/l1.sh"
|
||||||
|
run_stage approval-policy "${CI_DIR}/approval-policy.sh"
|
||||||
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
|
||||||
run_stage conformance "${CI_DIR}/conformance.sh"
|
run_stage conformance "${CI_DIR}/conformance.sh"
|
||||||
|
|
||||||
@@ -190,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
|
|||||||
pass) ok "spec-immutability: green" ;;
|
pass) ok "spec-immutability: green" ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
|
||||||
|
# is the approver, per operator intent. This stage is the single exception: a change to the
|
||||||
|
# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
|
||||||
|
# cannot honestly be approved BY the thing it changes, so those paths still need a human.
|
||||||
|
#
|
||||||
|
# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
|
||||||
|
# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
|
||||||
|
# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
|
||||||
|
case "${OUTCOME[approval-policy]}" in
|
||||||
|
fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
|
||||||
|
skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
|
||||||
|
pass) ok "approval-policy: green" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
|
||||||
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
|
||||||
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
---
|
||||||
|
date: 2026-09-30
|
||||||
|
branch: fix/flux-deploy-notification-inhibited
|
||||||
|
title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
|
||||||
|
deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
|
||||||
|
|
||||||
|
- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
|
||||||
|
**no `severity` label at all**, where it previously carried `severity: info`.
|
||||||
|
- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
|
||||||
|
`notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
|
||||||
|
- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
|
||||||
|
blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
|
||||||
|
|
||||||
|
**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
|
||||||
|
ships:
|
||||||
|
|
||||||
|
InfoInhibitor:
|
||||||
|
group by (namespace) (ALERTS{severity="info"} == 1)
|
||||||
|
unless on (namespace) group by (namespace)
|
||||||
|
(ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
|
||||||
|
|
||||||
|
which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
|
||||||
|
it**. The chart's third `inhibit_rule` then drops exactly that alert:
|
||||||
|
|
||||||
|
source_matchers: [alertname="InfoInhibitor"]
|
||||||
|
target_matchers: [severity="info"]
|
||||||
|
equal: [namespace]
|
||||||
|
|
||||||
|
For a deploy notification this is the worst possible interaction, because the suppressed case is the
|
||||||
|
*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
|
||||||
|
or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
|
||||||
|
delivered only when something else in `flux-system` was already broken.
|
||||||
|
|
||||||
|
Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
|
||||||
|
real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
|
||||||
|
label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
|
||||||
|
`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
|
||||||
|
around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
|
||||||
|
the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
|
||||||
|
— because the next reader's instinct will be to add one back.
|
||||||
|
|
||||||
|
## What I verified
|
||||||
|
|
||||||
|
Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
|
||||||
|
|
||||||
|
- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
|
||||||
|
`kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
|
||||||
|
populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
|
||||||
|
`status.artifact.revision` paths are correct — previously UNVERIFIED.
|
||||||
|
- **The `podAnnotations` roll mechanism works.** The KSM pod carries
|
||||||
|
`homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
|
||||||
|
(`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
|
||||||
|
genuinely closed rather than merely documented.
|
||||||
|
- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
|
||||||
|
converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
|
||||||
|
`revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
|
||||||
|
**cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
|
||||||
|
notification's whole mechanism.
|
||||||
|
- **The routing works.** `amtool config routes` shows
|
||||||
|
`{notification="deploy"} → receiver: email-deploy` under the default route, and
|
||||||
|
`amtool config routes test notification=deploy` returns `email-deploy`.
|
||||||
|
- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
|
||||||
|
the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
|
||||||
|
Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
|
||||||
|
is 0 for every reason, so nothing failed to send; it was never dispatched.
|
||||||
|
- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
|
||||||
|
|
||||||
|
## What I did NOT verify
|
||||||
|
|
||||||
|
- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
|
||||||
|
suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
|
||||||
|
(`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
|
||||||
|
proven by merging this and watching the next commit produce `[FLUX] applying` and then
|
||||||
|
`[FLUX] applied`. Same structural gap as PR #25, one layer further along.
|
||||||
|
- **Whether the single email during the window was ours.**
|
||||||
|
`alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
|
||||||
|
(`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
|
||||||
|
label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
|
||||||
|
attribution is genuinely unavailable from this metric.
|
||||||
|
- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
|
||||||
|
inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
|
||||||
|
apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
|
||||||
|
- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
|
||||||
|
`firing`/`resolved` in the notification template data.
|
||||||
|
- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
|
||||||
|
reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
|
||||||
|
model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
|
||||||
|
this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
|
||||||
|
- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
|
||||||
|
10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
{
|
||||||
|
"_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
|
||||||
|
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
|
||||||
|
"_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
|
||||||
|
"sensitive_paths": [
|
||||||
|
"ci/**",
|
||||||
|
".gitea/**",
|
||||||
|
"conformance/**",
|
||||||
|
"clusters/**",
|
||||||
|
"bootstrap/**",
|
||||||
|
"infra/gitea/**",
|
||||||
|
"platform/kyverno/**",
|
||||||
|
"platform-config/kustomization.yaml",
|
||||||
|
".sops.yaml",
|
||||||
|
"**/*.sops.yaml",
|
||||||
|
"**/rbac.yaml",
|
||||||
|
"CLAUDE.md"
|
||||||
|
],
|
||||||
|
"_why_each": {
|
||||||
|
"ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
|
||||||
|
"conformance/**": "the tests and the harness; changing them changes what green MEANS",
|
||||||
|
"clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
|
||||||
|
"infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
|
||||||
|
"platform/kyverno/**": "the admission policy engine",
|
||||||
|
".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
|
||||||
|
"CLAUDE.md": "the delivery contract agents follow"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -839,6 +839,14 @@ spec:
|
|||||||
# that has never applied anything — or a GitRepository with no artifact yet — publishes the
|
# that has never applied anything — or a GitRepository with no artifact yet — publishes the
|
||||||
# series with no `revision` label. Without the matcher those group together under the empty
|
# series with no `revision` label. Without the matcher those group together under the empty
|
||||||
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
|
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
|
||||||
|
#
|
||||||
|
# A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
|
||||||
|
# because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
|
||||||
|
# pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
|
||||||
|
# InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
|
||||||
|
# it is the only thing firing in its namespace, which is every healthy deploy. Measured on
|
||||||
|
# prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
|
||||||
|
# was inhibited in between. The full reasoning is on the `notification: deploy` label below.
|
||||||
# ===================================================================================
|
# ===================================================================================
|
||||||
- alert: FluxRolloutInProgress
|
- alert: FluxRolloutInProgress
|
||||||
expr: |
|
expr: |
|
||||||
@@ -857,7 +865,31 @@ spec:
|
|||||||
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that
|
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that
|
||||||
# only sometimes arrives is worse than no notification at all.
|
# only sometimes arrives is worse than no notification at all.
|
||||||
labels:
|
labels:
|
||||||
severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied"
|
# ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
|
||||||
|
# This alert originally used `severity: info`. That was WRONG, and wrong in the silent
|
||||||
|
# direction — measured on prod within minutes of the first real deploy.
|
||||||
|
# kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
|
||||||
|
# group by (namespace) (ALERTS{severity="info"} == 1)
|
||||||
|
# unless on (namespace) group by (namespace)
|
||||||
|
# (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
|
||||||
|
# which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
|
||||||
|
# it; paired with the chart's inhibit_rule
|
||||||
|
# source_matchers: [alertname="InfoInhibitor"]
|
||||||
|
# target_matchers: [severity="info"]
|
||||||
|
# equal: [namespace]
|
||||||
|
# the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
|
||||||
|
# happening in its namespace. For a deploy notification that is the only case that
|
||||||
|
# matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
|
||||||
|
# critical alert, so InfoInhibitor fired and this alert never reached the receiver.
|
||||||
|
# Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
|
||||||
|
# alert began firing for 12 Kustomizations.
|
||||||
|
# Upstream's intent is explicit — `severity: info` means "only worth showing next to a
|
||||||
|
# real problem". A deploy notice is not that, so it must not claim that severity.
|
||||||
|
# `notification` is a label no inhibit_rule in the chart targets (all three match on
|
||||||
|
# severity values), and omitting `severity` also stops InfoInhibitor firing for
|
||||||
|
# flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
|
||||||
|
# `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
|
||||||
|
notification: deploy
|
||||||
annotations:
|
annotations:
|
||||||
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
|
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
|
||||||
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
|
||||||
|
|||||||
@@ -139,11 +139,19 @@ spec:
|
|||||||
#
|
#
|
||||||
# A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
|
# A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
|
||||||
# routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
|
# routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
|
||||||
# real platform fault, which is how alert mail stops being read at all. `severity: info`
|
# real platform fault, which is how alert mail stops being read at all.
|
||||||
# is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
|
#
|
||||||
# route claims exactly the deploy notices and nothing else. Keep it that way — giving a
|
# MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
|
||||||
# genuine fault severity: info would silently route it here and re-label it a deploy.
|
# `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
|
||||||
- matchers: ['severity = "info"']
|
# was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
|
||||||
|
# info alert with no warning/critical beside it, and its inhibit_rule
|
||||||
|
# (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
|
||||||
|
# exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
|
||||||
|
# so the notification was inhibited on every occasion it was wanted. Measured live on
|
||||||
|
# prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
|
||||||
|
# Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
|
||||||
|
# which match on severity values only. See the label's own comment in alerts.yaml.
|
||||||
|
- matchers: ['notification = "deploy"']
|
||||||
receiver: email-deploy
|
receiver: email-deploy
|
||||||
# One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
|
# One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
|
||||||
# Kustomization (16 possible), and grouping on alertname alone collapses them into a
|
# Kustomization (16 possible), and grouping on alertname alone collapses them into a
|
||||||
|
|||||||
@@ -89,7 +89,7 @@ tests:
|
|||||||
alertname: FluxRolloutInProgress
|
alertname: FluxRolloutInProgress
|
||||||
exp_alerts:
|
exp_alerts:
|
||||||
- exp_labels:
|
- exp_labels:
|
||||||
severity: info
|
notification: deploy
|
||||||
job: kube-state-metrics
|
job: kube-state-metrics
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
name: platform
|
name: platform
|
||||||
@@ -148,7 +148,7 @@ tests:
|
|||||||
alertname: FluxRolloutInProgress
|
alertname: FluxRolloutInProgress
|
||||||
exp_alerts:
|
exp_alerts:
|
||||||
- exp_labels:
|
- exp_labels:
|
||||||
severity: info
|
notification: deploy
|
||||||
job: kube-state-metrics
|
job: kube-state-metrics
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
name: platform
|
name: platform
|
||||||
@@ -229,7 +229,7 @@ tests:
|
|||||||
alertname: FluxRolloutInProgress
|
alertname: FluxRolloutInProgress
|
||||||
exp_alerts:
|
exp_alerts:
|
||||||
- exp_labels:
|
- exp_labels:
|
||||||
severity: info
|
notification: deploy
|
||||||
job: kube-state-metrics
|
job: kube-state-metrics
|
||||||
namespace: flux-system
|
namespace: flux-system
|
||||||
name: conformance
|
name: conformance
|
||||||
|
|||||||
Reference in new issue
Block a user