fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret #60

Merged

jakman85 merged 2 commits from fix/vpn-server-key-in-sops into main 2026-10-06 06:09:05 +00:00

Conversation 1 Commits 2 Files Changed 14

+584 -90

gitea-admin commented 2026-10-06 04:56:03 +00:00

Owner

Copy link

Copy source

You were right. #55 shipped the WireGuard server private key as a Secret to kubectl create by hand. This moves it into the tree as platform/vpn-ingress/wg-server-key.sops.yaml, where every other secret in this repo lives.

Why the original shape was wrong, not just unconventional

1. R-SEC-01 / spec docs/07 — secrets live in source, SOPS-encrypted, reaching the cluster only through Flux. An out-of-band Secret is outside the GitOps tree by construction.
2. It broke DR for the one credential that cannot be re-derived. A rebuild restores everything in git; it does not restore a hand-made Secret. Losing this key means re-enrolling every device, because each client profile pins its public half. #55 papered over that with "keep a copy in your password manager" — a workaround for a self-inflicted gap.

Worth recording: platform/kyverno/admission/secret-provenance.yaml did not forbid the original shape. It scopes to app-namespace=true, and vpn-ingress is deliberately a platform namespace, so admission would have accepted the hand-applied Secret. The rule broken was the written requirement, not an enforced control — "the gate let it through" was never evidence it was correct.

How the unfillable-key problem is solved

An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed already encrypted, with privatekey: REPLACE-ME. That makes the wiring complete and green from the first commit — resource line plus both overlay decryption patches — leaving you exactly one act: replace the value.

A commented-out resource line to uncomment later was the alternative, and it is precisely the "in git, no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by (alert-runbooks-resolvable exists because of it).

The init guard is now a validation, not an emptiness check. The value is piped through wg pubkey, which accepts only a real curve25519 private key, and the pod names which case it hit:

pod log says	meaning
REPLACE-ME PLACEHOLDER	expected fresh-install state
It is EMPTY	blanked value
still SOPS CIPHERTEXT (starts with ENC[)	an overlay is missing its decryption patch — this applies cleanly with every Flux condition green, and is the silent failure the guard exists for
rejected, none of the above	mangled value, or a public key pasted by mistake

A second finding — and my first fix for it was also wrong

wg existed only on the host, sops only in the container, qrencode on neither. So sops -d … | wg pubkey could not run in one shell, and the command #55 documented could not have worked.

My first fix was to split every step across the boundary and tell the operator to dnf install qrencode on their workstation. Operator correction: tool requirements are supposed to be baked into the devcontainer — CLAUDE.md §5 and the devcontainer README's "Don't install the toolchain on the host". The split honoured the symptom and broke the invariant, and produced a procedure that cannot be reproduced in CI.

wireguard-tools + qrencode are now distro packages in infra/devcontainer/Containerfile. Both are in Fedora 44's repos (wireguard-tools 1.0.20260223 — the same upstream version as the container image this component runs — and qrencode 4.1.1), unpinned like every other dnf package there: the pinned thing is the Fedora base, and neither is a gate tool whose version changes what green means.

Three things got simpler rather than merely relocated, which is the tell that the rule was right:

1. Step 2 is one pipeline in one shell — sops set … "$(wg genkey)" writes the key straight into the already-encrypted document. Gone: the staged .cache/server.key copy, the warning that only the *.key suffix protected it, and the sha256 comparison to prove the right bytes landed.
2. new-peer.sh derives the server public key from wg-server-key.sops.yaml itself (sops -d --extract | wg pubkey), so there is no local key copy to keep in step with the encrypted one, and no magic host path it has to live at.
3. .cache/ is gitignored at the repo root (mirroring infra/ansible/.gitignore), giving the container a persistent, non-committable scratch dir. new-peer.sh writes profiles to .cache/wg-peers/ — necessary because the repo bind-mount is the container's only persistent read-write mount, so a profile written to /tmp in there is gone before it can reach a desktop. Its blanket "refuse to write inside the checkout" guard is now a git check-ignore test of the actual property.

Verified by execution rather than reasoning: the image rebuilds; wg / qrencode / sops all resolve in it; sops -d | wg pubkey runs in one shell; new-peer.sh refuses the REPLACE-ME placeholder with an actionable message; and a full run with a throwaway server key renders a correct profile at 0600 under a 0700 gitignored dir, prints the QR, and refuses a non-ignored destination inside the repo.

The gate caught three real defects in this branch, including one in the fix for another

Recorded because this is the gate earning its keep:

1. no-plaintext-secrets FAILED — my kustomization named the .example template in a comment, and the check matches comments too (correctly: naming one is how a plaintext Secret gets applied by accident). platform/crafty/kustomization.yaml never names its own example; the convention was there to copy and I missed it. I had also asserted in my handoff entry that this check passed, without running it. It did not. That claim is corrected in the entry.
2. flux-postbuild-substitution FAILED on ${key#ENC[} in the new guard, flagged [SILENT-EMPTY] — envsubst consumes every dollar-brace expansion, so my prefix-strip would have rendered empty and silently disabled the ciphertext branch of the guard, which is exactly the failure the guard was added to catch. Rewritten with case.
3. Then it failed a third time, on the fix's own comment — [FATAL-PARSE], the prose explaining the rule, written with the literal form. That class fails the entire Kustomization, not just the line. Second time in this component that typing the hazard inside a comment about the hazard has been the defect, so: in a substituted path, describe the construct in words, never type it.

After all three, l0 is green. Gate: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two environmental skips. Not green — reported as rc 3.

What is NOT verified

That Flux decrypts this on prod. The overlay patches have never reconciled, and this is the whole risk. It has one observable outcome after merge: the pods should CrashLoop reporting the REPLACE-ME placeholder (decryption worked), not ENC[ (a patch is wrong). I built the guard to tell those apart precisely because I cannot test it.

Also unverified: wg pubkey through this path (wg is absent from both the container and the gate, so its first run will be in the pod), and the sops set command's nested quoting inside dev.sh run bash -c — the editor form sops <file> is the quoting-independent fallback.

Sensitive paths

Touches clusters/** (both overlays) and adds a **/*.sops.yaml, so ci/approval-policy.sh will BLOCK pending your /approve-override <reason> as a PR review of type "Comment". Deliberately not posted by me (CLAUDE.md §1).

🤖 Generated with Claude Code

**You were right.** #55 shipped the WireGuard server private key as a Secret to `kubectl create` by hand. This moves it into the tree as `platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives. ## Why the original shape was wrong, not just unconventional 1. **R-SEC-01 / spec docs/07** — secrets live in source, SOPS-encrypted, reaching the cluster only through Flux. An out-of-band Secret is outside the GitOps tree by construction. 2. **It broke DR for the one credential that cannot be re-derived.** A rebuild restores everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling every device**, because each client profile pins its public half. #55 papered over that with "keep a copy in your password manager" — a workaround for a self-inflicted gap. Worth recording: `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape. It scopes to `app-namespace=true`, and `vpn-ingress` is deliberately a platform namespace, so admission would have accepted the hand-applied Secret. The rule broken was the written requirement, not an enforced control — "the gate let it through" was never evidence it was correct. ## How the unfillable-key problem is solved An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That makes the wiring complete and green from the first commit — resource line plus both overlay decryption patches — leaving you exactly one act: replace the value. A commented-out resource line to uncomment later was the alternative, and it is precisely the *"in git, no generator entry: reviewed, merged, apparently done, still broken"* shape this repo keeps getting bitten by (`alert-runbooks-resolvable` exists because of it). **The init guard is now a validation, not an emptiness check.** The value is piped through `wg pubkey`, which accepts only a real curve25519 private key, and the pod names which case it hit: | pod log says | meaning | |---|---| | `REPLACE-ME PLACEHOLDER` | expected fresh-install state | | `It is EMPTY` | blanked value | | **`still SOPS CIPHERTEXT (starts with ENC[)`** | **an overlay is missing its decryption patch** — this applies cleanly with every Flux condition green, and is the silent failure the guard exists for | | rejected, none of the above | mangled value, or a *public* key pasted by mistake | ## A second finding — and my first fix for it was also wrong `wg` existed only on the **host**, `sops` only in the **container**, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and **the command #55 documented could not have worked.** My first fix was to split every step across the boundary and tell the operator to `dnf install qrencode` on their workstation. **Operator correction: tool requirements are supposed to be baked into the devcontainer** — CLAUDE.md §5 and the devcontainer README's "Don't install the toolchain on the host". The split honoured the symptom and broke the invariant, and produced a procedure that cannot be reproduced in CI. **`wireguard-tools` + `qrencode` are now distro packages in `infra/devcontainer/Containerfile`.** Both are in Fedora 44's repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there: the pinned thing is the Fedora base, and neither is a gate tool whose version changes what green means. Three things got **simpler** rather than merely relocated, which is the tell that the rule was right: 1. **Step 2 is one pipeline in one shell** — `sops set … "$(wg genkey)"` writes the key straight into the already-encrypted document. Gone: the staged `.cache/server.key` copy, the warning that only the `*.key` suffix protected it, and the sha256 comparison to prove the right bytes landed. 2. **`new-peer.sh` derives the server public key from `wg-server-key.sops.yaml` itself** (`sops -d --extract | wg pubkey`), so there is no local key copy to keep in step with the encrypted one, and no magic host path it has to live at. 3. **`.cache/` is gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), giving the container a persistent, non-committable scratch dir. `new-peer.sh` writes profiles to `.cache/wg-peers/` — necessary because the repo bind-mount is the container's *only* persistent read-write mount, so a profile written to `/tmp` in there is gone before it can reach a desktop. Its blanket "refuse to write inside the checkout" guard is now a `git check-ignore` test of the actual property. Verified by execution rather than reasoning: the image rebuilds; `wg` / `qrencode` / `sops` all resolve in it; `sops -d | wg pubkey` runs in one shell; `new-peer.sh` refuses the `REPLACE-ME` placeholder with an actionable message; and a full run with a throwaway server key renders a correct profile at `0600` under a `0700` gitignored dir, prints the QR, and refuses a non-ignored destination inside the repo. ## The gate caught three real defects in this branch, including one in the fix for another Recorded because this is the gate earning its keep: 1. **`no-plaintext-secrets` FAILED** — my kustomization named the `.example` template *in a comment*, and the check matches comments too (correctly: naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml` never names its own example; the convention was there to copy and I missed it. **I had also asserted in my handoff entry that this check passed, without running it. It did not.** That claim is corrected in the entry. 2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in the new guard, flagged `[SILENT-EMPTY]` — envsubst consumes every dollar-brace expansion, so my prefix-strip would have rendered empty and **silently disabled the ciphertext branch of the guard**, which is exactly the failure the guard was added to catch. Rewritten with `case`. 3. **Then it failed a third time, on the fix's own comment** — `[FATAL-PARSE]`, the prose explaining the rule, written with the literal form. That class fails the **entire** Kustomization, not just the line. Second time in this component that typing the hazard inside a comment about the hazard has been the defect, so: in a substituted path, describe the construct in words, never type it. After all three, `l0` is green. Gate: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green — reported as rc 3.** ## What is NOT verified **That Flux decrypts this on prod.** The overlay patches have never reconciled, and this is the whole risk. It has one observable outcome after merge: the pods should CrashLoop reporting the **`REPLACE-ME` placeholder** (decryption worked), *not* **`ENC[`** (a patch is wrong). I built the guard to tell those apart precisely because I cannot test it. Also unverified: `wg pubkey` through this path (`wg` is absent from both the container and the gate, so its first run will be in the pod), and the `sops set` command's nested quoting inside `dev.sh run bash -c` — the editor form `sops <file>` is the quoting-independent fallback. ## Sensitive paths Touches `clusters/**` (both overlays) and adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will **BLOCK** pending your `/approve-override <reason>` as a PR **review** of type "Comment". Deliberately not posted by me (CLAUDE.md §1). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-06 04:56:03 +00:00

fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret ... e35612dc7e

Operator correction: secrets are kept in source through sops. #55 shipped the
WireGuard server private key as a Secret to `kubectl create` by hand. Two real
costs, not just a convention break:

  * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and
    reach the cluster only through Flux.
  * It broke DR for the one credential that cannot be re-derived. A rebuild
    restores everything in git; it does not restore a hand-made Secret, and
    losing this key means re-enrolling EVERY device, because each client
    profile pins its public half. #55 papered over that with "keep a copy in
    your password manager".

(Note secret-provenance did NOT forbid the original shape -- it scopes to
app-namespace=true and vpn-ingress is deliberately a platform namespace. The
rule broken was the written requirement, not an enforced control.)

An agent still must not mint a long-lived private key (CLAUDE.md §6), so the
file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes
the wiring complete and green from the first commit -- resource line plus both
overlay decryption patches -- leaving the operator exactly one act: replace the
value. A commented-out resource to uncomment later is the "in git, no generator
entry, reviewed, merged, still broken" shape this repo keeps hitting.

The init guard is now a validation rather than an emptiness check: the value is
piped through `wg pubkey`, which accepts only a real curve25519 private key, and
the pod names which case it hit -- placeholder, empty, mangled, or (the silent
one) still `ENC[` ciphertext because an overlay is missing its decryption patch,
which otherwise applies cleanly with every Flux condition green.

Second, independent finding that also fixes #55's step 2: `wg` is on the HOST
only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in
one shell -- the command #55 documented could not have worked. Every step is now
split by where it runs, and the written key is verified by comparing sha256
digests rather than printing anything. `qrencode` is on neither, so the QR step
says so and gives the install command.

Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block
pending the operator's /approve-override. Deliberately not posted by me.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin added 1 commit 2026-10-06 05:50:11 +00:00

fix(devcontainer): bake wg + qrencode into the image instead of splitting the VPN procedure across host and container ... 8059297036

Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."

My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.

wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.

Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:

  * Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
    key straight into the already-encrypted document. No staged copy in .cache, no
    `*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
  * new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
    (`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
    step with the encrypted one, and no magic host path it must live at.
  * `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
    giving the container a persistent non-committable scratch dir. new-peer.sh
    writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
    the container's only persistent read-write mount, so a profile in /tmp there is
    gone before it can reach a desktop. Its blanket "refuse to write inside the
    checkout" guard is now a `git check-ignore` test of the real property.

Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes 2026-10-06 05:53:45 +00:00

jakman85 merged commit 4791653f2b into main 2026-10-06 06:09:05 +00:00

jakman85 referenced this issue from a commit 2026-10-06 06:09:06 +00:00

Merge pull request 'fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret' (#60) from fix/vpn-server-key-in-sops into main

gitea-admin referenced this pull request 2026-10-06 07:29:00 +00:00

fix(vpn-ingress): new-peer.sh counted its own commented-out example as an allocated address #64

Sign in to join this conversation.

Reviewers

No Reviewers

jakman85

Labels

Clear labels

agent

automated-update

blast-radius

breaking

ci-tooling

dependencies

needs-review

own-image

range-widen

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#60