fix(node): pin gitea to the Gateway VIP — the failing certificate was the router's, not the Gateway's #59

Merged

jakman85 merged 4 commits from docs/pm-agent-registry-pull-prereq into main 2026-10-06 05:51:22 +00:00

Conversation 0 Commits 4 Files Changed 9

+378 -10

gitea-admin commented 2026-10-06 04:24:48 +00:00

Owner

Copy link

Copy source

Three rounds of operator review on #57, in one branch. The headline is the last one.

The image pull: diagnosed, and it was never the image

containerd pulls as a node process, so it resolves through node_baseline_dns_servers
(the LAN gateway, deliberately not Pi-hole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev → 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8.
This router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured from the workstation with the right SNI:

subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
No extensions in certificate

No extensions means no subjectAltName, which is exactly x509: certificate is not valid for any names. Meanwhile 192.168.2.20 is the shared Gateway's own VIP and its https
listener serves the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard
certificate and the digest were never implicated — only the node's answer was wrong.

This repository had already measured the identical failure for the Keycloak issuer from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18) and already carries the
mechanism that fixes it. gitea is simply the second node process to need it, and nobody
had added it:

node_baseline_hosts_pins:
  - ip: "192.168.2.20"
    names:
      - "keycloak.{{ k3s_base_domain }}"
      - "gitea.{{ k3s_base_domain }}"      # <- this

An operator must run the node_baseline play — writing /etc/hosts on a node is not
something Flux or an agent can do. Parity checked against the live cluster, not the comments:
BASE_DOMAIN=home.keithjackson.dev, GATEWAY_VIP=192.168.2.20 (the file's note about prod
serving prod.home.keithjackson.dev is stale history).

Gitea auth: local account + PAT, recorded as a decision

Written into the README and the credential hand-off so it is not re-litigated, with the
reason the federated alternative is not merely unchosen but unavailable: Gitea's API
accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens, and as an OAuth2
provider Gitea implements only the Authorization Code Grant — no client-credentials
grant, and nothing that validates a third-party OIDC token. A Keycloak service-account token
cannot authenticate to Gitea's API or to git-over-HTTP. Keycloak stays what it is for humans:
the browser login source on Gitea.

What the hand-off still enforces: the account must not be an administrator, because
approval-policy.json authorises /approve-override on live repository permission alone.

The agent is IN the cluster: one git host

pm-worker's host allowlist accepted two values; the external hostname is removed. In-cluster
consumers reach a service by its *.svc.cluster.local name — a different path by design from
how an outside client reaches it — and as an accepted value the external name widened the
allowlist to a host reachable over the namespace's world:443 rule, the exact injection shape
the allowlist exists to refuse. probe-validation.sh pins it: 20 refusals.

Post-merge verification of #57 (unchanged from the first push)

Flux applied main@sha1:95597bd… to platform-config, pm-agent and apps-wave-0, all
Ready. ClusterPolicy/pm-agent-job-shape is live, Enforce + failurePolicy: Fail, all
six rules — the check that mattered most, since a policy in the wrong kustomization list
ships green and never reaches the cluster. SOPS decryption works through the new overlay patch
(key names and value lengths only, never values). pm-agent-worker is bound to nothing.

Still open

Registry authentication. Unauthenticated clients get nothing from this registry and no
imagePullSecret exists in the tree. Consistent with the credential decision, the shape is a
dockerconfigjson for a local Gitea account whose token carries read:package — not built,
because minting it is yours, and because the worker Job template's shape is admission-pinned
so whether the policy should also pin imagePullSecrets deserves a deliberate look.

Also flagged honestly in the handoff entry: gitea.home.keithjackson.dev resolves publicly
to the WAN CNAME even though catalog/gitea.yaml inherits exposure: internal, which is
supposed to publish no public record. I did not chase it — it did not change the diagnosis —
but "internal services publish no public DNS" is an L0-checked claim and this looks like a
case worth someone's attention.

Gate

./ci/gate.sh in the pinned toolbox from a §0 worktree → exit 3 (UNDETERMINED),
gitea-settings the only stage that did not run (no SETTINGS_API_TOKEN on the workstation);
every other stage green, spec-immutability included. rc 3 is not a pass. No sensitive path.

🤖 Generated with Claude Code

Three rounds of operator review on #57, in one branch. The headline is the last one. ## The image pull: diagnosed, and it was never the image containerd pulls as a **node** process, so it resolves through `node_baseline_dns_servers` (the LAN gateway, deliberately not Pi-hole) and gets the public answer: a CNAME to `wan.home.keithjackson.dev` → `136.51.115.252`, measured via both `1.1.1.1` and `8.8.8.8`. This router does not hairpin, so WAN:443 answers with its own placeholder certificate — measured from the workstation with the right SNI: ``` subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D No extensions in certificate ``` No extensions means no `subjectAltName`, which is exactly `x509: certificate is not valid for any names`. Meanwhile `192.168.2.20` **is** the shared Gateway's own VIP and its `https` listener serves the real Let's Encrypt `*.home.keithjackson.dev`. **The Gateway, the wildcard certificate and the digest were never implicated — only the node's answer was wrong.** This repository had already measured the identical failure for the Keycloak issuer from k3s-prod-2/-3 on 2026-10-03 (`curl` exit 60, `ssl_verify_result=18`) and already carries the mechanism that fixes it. `gitea` is simply the second node process to need it, and nobody had added it: ```yaml node_baseline_hosts_pins: - ip: "192.168.2.20" names: - "keycloak.{{ k3s_base_domain }}" - "gitea.{{ k3s_base_domain }}" # <- this ``` **An operator must run the `node_baseline` play** — writing `/etc/hosts` on a node is not something Flux or an agent can do. Parity checked against the live cluster, not the comments: `BASE_DOMAIN=home.keithjackson.dev`, `GATEWAY_VIP=192.168.2.20` (the file's note about prod serving `prod.home.keithjackson.dev` is stale history). ## Gitea auth: local account + PAT, recorded as a decision Written into the README and the credential hand-off so it is not re-litigated, with the reason the federated alternative is not merely unchosen but **unavailable**: Gitea's API accepts basic auth, PATs, SSH signatures and its *own* OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization Code Grant** — no client-credentials grant, and nothing that validates a third-party OIDC token. A Keycloak service-account token cannot authenticate to Gitea's API or to git-over-HTTP. Keycloak stays what it is for humans: the browser login source on Gitea. What the hand-off still enforces: **the account must not be an administrator**, because `approval-policy.json` authorises `/approve-override` on live repository permission alone. ## The agent is IN the cluster: one git host `pm-worker`'s host allowlist accepted two values; the external hostname is removed. In-cluster consumers reach a service by its `*.svc.cluster.local` name — a different path by design from how an outside client reaches it — and as an *accepted* value the external name widened the allowlist to a host reachable over the namespace's `world:443` rule, the exact injection shape the allowlist exists to refuse. `probe-validation.sh` pins it: **20 refusals**. ## Post-merge verification of #57 (unchanged from the first push) Flux applied `main@sha1:95597bd…` to `platform-config`, `pm-agent` and `apps-wave-0`, all Ready. **`ClusterPolicy/pm-agent-job-shape` is live, `Enforce` + `failurePolicy: Fail`, all six rules** — the check that mattered most, since a policy in the wrong kustomization list ships green and never reaches the cluster. SOPS decryption works through the new overlay patch (key names and value lengths only, never values). `pm-agent-worker` is bound to nothing. ## Still open **Registry authentication.** Unauthenticated clients get nothing from this registry and no `imagePullSecret` exists in the tree. Consistent with the credential decision, the shape is a `dockerconfigjson` for a local Gitea account whose token carries `read:package` — not built, because minting it is yours, and because the worker Job template's shape is admission-pinned so whether the policy should also pin `imagePullSecrets` deserves a deliberate look. Also flagged honestly in the handoff entry: `gitea.home.keithjackson.dev` resolves **publicly** to the WAN CNAME even though `catalog/gitea.yaml` inherits `exposure: internal`, which is supposed to publish no public record. I did not chase it — it did not change the diagnosis — but "internal services publish no public DNS" is an L0-checked claim and this looks like a case worth someone's attention. ## Gate `./ci/gate.sh` in the pinned toolbox from a §0 worktree → **exit 3 (UNDETERMINED)**, `gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on the workstation); every other stage green, `spec-immutability` included. rc 3 is not a pass. No sensitive path. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-06 04:24:48 +00:00

docs(pm-agent): the registry cannot be pulled from yet — name both prerequisites where the operator reads them ... 6253d546f8

pm-agent reconciled exactly as designed after #57: Flux applied main@sha1:95597bd to
platform-config, pm-agent and apps-wave-0 (all Ready), the ClusterPolicy
pm-agent-job-shape is live with Enforce + failurePolicy Fail and all six rules, SOPS
decryption works through the new overlay patch (checked by key names and value lengths,
never values), and pm-agent-worker really is bound to nothing.

Measuring it turned up two prerequisites for the NEXT operator step that nothing in this
repository declares, and that #57's handoff could not have known — it listed the image
pull as unverified, and that is exactly where it bit:

 - the nodes cannot verify the registry's TLS certificate (containerd: 'x509: certificate
   is not valid for any names'), while the same name serves a healthy Let's Encrypt
   wildcard from the LAN. A correct digest fails identically;
 - an unauthenticated client gets nothing from this Gitea's registry (its anonymous token
   carries no access scope) and no imagePullSecret exists anywhere in the tree.

Not specific to pm-agent: conformance-runner shares the registry path and has never been
pulled either. The dispatcher is the first workload in the cluster ever to reference it.

Documentation only — no manifest, no policy, no code. Both fixes need an operator action
and one of them is a posture decision, so the docs present both rather than pick.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin added 1 commit 2026-10-06 04:42:24 +00:00

fix(pm-agent): the agent is IN the cluster — one git host, the cluster-local service name ... a209a618fd

An in-cluster consumer reaches a service by its *.svc.cluster.local name, which is by
design a different path from how a client outside the cluster reaches the same service.
pm-worker's host allowlist accepted TWO values: the cluster-local name and the external
hostname gitea.home.keithjackson.dev. The second is removed.

It was never needed — nothing sets it — it would route in-cluster traffic out through the
Gateway and back for no reason, and as an ACCEPTED value it widened the allowlist to a
host reachable over the namespace's world:443 egress rule, which is precisely the
injection shape the allowlist exists to refuse.

probe-validation.sh now pins that: 20 refusals, including the external hostname, the
cluster-local name over https, and the cluster-local name on the wrong port.

Also corrects a remark in the handoff entry: I cited cluster-vars' GATEWAY_VIP
(192.168.1.53) as suspicious next to the nodes' 192.168.2.x. That was noise — those base
cluster-vars values are dev-era leftovers for a decommissioned cluster and say nothing
about prod. None of the measured facts depended on it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin changed title from docs(pm-agent): the registry cannot be pulled from yet — name both prerequisites where the operator reads them to fix(pm-agent): one git host — the cluster-local service name; plus the post-merge registry findings 2026-10-06 04:42:49 +00:00

gitea-admin added 1 commit 2026-10-06 05:36:54 +00:00

fix(node): pin gitea to the Gateway VIP — containerd was resolving it to the non-hairpinned WAN ... 523a463fd7

The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.

containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:

    subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
    No extensions in certificate

No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.

This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.

Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin changed title from fix(pm-agent): one git host — the cluster-local service name; plus the post-merge registry findings to fix(node): pin gitea to the Gateway VIP — the failing certificate was the router's, not the Gateway's 2026-10-06 05:37:25 +00:00

gitea-admin added 1 commit 2026-10-06 05:45:57 +00:00

docs(node): name the exact Ansible command that applies the pin, not 'run the play' ... 90a79d29d3

Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.

The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:

    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
      -e k3s_hosts=k3s_prod --tags hosts

--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.

Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit 523f482957 into main 2026-10-06 05:51:22 +00:00

jakman85 referenced this issue from a commit 2026-10-06 05:51:24 +00:00

Merge pull request 'fix(node): pin gitea to the Gateway VIP — the failing certificate was the router's, not the Gateway's' (#59) from docs/pm-agent-registry-pull-prereq into main

gitea-admin referenced this issue from a commit 2026-10-06 06:03:55 +00:00

feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read

gitea-admin referenced this pull request 2026-10-06 06:04:20 +00:00

feat(pm-agent): pull credential + admission rule 7, first published digest pinned, dry-run first start, token scopes fixed #61

gitea-admin referenced this issue from a commit 2026-10-30 06:19:51 +00:00

feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

agent

automated-update

blast-radius

breaking

ci-tooling

dependencies

needs-review

own-image

range-widen

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#59