docs(node): name the exact Ansible command that applies the pin, not 'run the play'
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.

The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:

    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
      -e k3s_hosts=k3s_prod --tags hosts

--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.

Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 00:44:26 -05:00
1 parent 523a463fd7
commit 90a79d29d3
4 files changed
+26 -8

No files matched your search

@@ -13,9 +13,21 @@ has.
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
Keycloak issuer pin that is there for the identical reason. **An operator must run the
`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not
something Flux or an agent can do.
Keycloak issuer pin that is there for the identical reason.
**It reaches the nodes the way every other node setting in this project does — through
Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
baseline setting. Running the play is an operator action (it has become/root on prod), but
the configuration itself is declared, reviewed and version-controlled — not typed on a node.
The docs that described this as an undiagnosed blocker now describe the cause and the fix:
`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
@@ -86,7 +98,10 @@ with `gitea-settings` the only stage that did not run.
## What I did NOT verify
- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
nodes is written by the `node_baseline` play, which is an operator action. Until that runs
nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
`state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
that it *will* is the Keycloak pin working for the same cause — which I also did not
re-measure.
+2 -1
View File
@@ -239,7 +239,8 @@ about runtime behaviour in this document is a design claim, not a measurement.
(i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
`gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
pin the Keycloak issuer already carries; an operator must run the `node_baseline` play;
pin the Keycloak issuer already carries, applied by
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
the credential decision now being local Gitea accounts with PATs, that means a
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+4 -2
View File
@@ -67,8 +67,10 @@ Both are deliberate, and both fail **visibly** rather than silently:
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it
to reach the nodes.**
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
inventory, never typed on a node. Apply it scoped to that one task:
`cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
--tags hosts`.
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
unauthenticated client gets nothing from this Gitea's registry, and there is no
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+1 -1
View File
@@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely.
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |