docs(node): name the exact Ansible command that applies the pin, not 'run the play'
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.

The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:

    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
      -e k3s_hosts=k3s_prod --tags hosts

--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.

Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 00:44:26 -05:00
1 parent 523a463fd7
commit 90a79d29d3
4 files changed
+26 -8

No files matched your search

@@ -13,9 +13,21 @@ has.
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
Keycloak issuer pin that is there for the identical reason. **An operator must run the Keycloak issuer pin that is there for the identical reason.
`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not
something Flux or an agent can do. **It reaches the nodes the way every other node setting in this project does — through
Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
baseline setting. Running the play is an operator action (it has become/root on prod), but
the configuration itself is declared, reviewed and version-controlled — not typed on a node.
The docs that described this as an undiagnosed blocker now describe the cause and the fix: The docs that described this as an undiagnosed blocker now describe the cause and the fix:
`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
@@ -86,7 +98,10 @@ with `gitea-settings` the only stage that did not run.
## What I did NOT verify ## What I did NOT verify
- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the - **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
nodes is written by the `node_baseline` play, which is an operator action. Until that runs nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
`state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
and an image exists, "the node now reaches the Gateway" is a design claim. The evidence and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
that it *will* is the Keycloak pin working for the same cause — which I also did not that it *will* is the Keycloak pin working for the same cause — which I also did not
re-measure. re-measure.
+2 -1
View File
@@ -239,7 +239,8 @@ about runtime behaviour in this document is a design claim, not a measurement.
(i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
`gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; pin the Keycloak issuer already carries, applied by
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
the credential decision now being local Gitea accounts with PATs, that means a the credential decision now being local Gitea accounts with PATs, that means a
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+4 -2
View File
@@ -67,8 +67,10 @@ Both are deliberate, and both fail **visibly** rather than silently:
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
to reach the nodes.** inventory, never typed on a node. Apply it scoped to that one task:
`cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
--tags hosts`.
- **The pull is still unauthenticated** once the node reaches the right endpoint. An - **The pull is still unauthenticated** once the node reaches the right endpoint. An
unauthenticated client gets nothing from this Gitea's registry, and there is no unauthenticated client gets nothing from this Gitea's registry, and there is no
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+1 -1
View File
@@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely.
|---|---|---| |---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |