From 90a79d29d340c50b22ffe59a3a13cb5d2186d4c8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 6 Oct 2026 00:44:26 -0500 Subject: [PATCH] docs(node): name the exact Ansible command that applies the pin, not 'run the play' MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Node configuration in this project is Ansible's — declared in the inventory, reviewed, version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an /etc/hosts pin. The pin was already in the right place; what the docs lacked was the command. All four places now carry the scoped, idempotent one: cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags hosts --tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other baseline setting. Also verified the mechanism is not inert, which is the failure mode this repo keeps meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked blockinfile with backup: true, and state: absent when the list empties — so a removed entry really does leave the node. Co-Authored-By: Claude Opus 5 (1M context) --- ...-06-pm-agent-node-resolves-gitea-to-wan.md | 23 +++++++++++++++---- docs/plans/pm-agent-issue-to-pr.md | 3 ++- platform/pm-agent/README.md | 6 +++-- runbooks/pm-agent.md | 2 +- 4 files changed, 26 insertions(+), 8 deletions(-) diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md index 623d7d1..45ae653 100644 --- a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -13,9 +13,21 @@ has. `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the -Keycloak issuer pin that is there for the identical reason. **An operator must run the -`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not -something Flux or an agent can do. +Keycloak issuer pin that is there for the identical reason. + +**It reaches the nodes the way every other node setting in this project does — through +Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs, +OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the +*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the +dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no +cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command: + + cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ + -e k3s_hosts=k3s_prod --tags hosts + +`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other +baseline setting. Running the play is an operator action (it has become/root on prod), but +the configuration itself is declared, reviewed and version-controlled — not typed on a node. The docs that described this as an undiagnosed blocker now describe the cause and the fix: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and @@ -86,7 +98,10 @@ with `gitea-settings` the only stage that did not run. ## What I did NOT verify - **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the - nodes is written by the `node_baseline` play, which is an operator action. Until that runs + nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator + action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml` + renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and + `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs and an image exists, "the node now reaches the Gateway" is a design claim. The evidence that it *will* is the Keycloak pin working for the same cause — which I also did not re-measure. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 567ac69..ad87805 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -239,7 +239,8 @@ about runtime behaviour in this document is a design claim, not a measurement. (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same - pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; + pin the Keycloak issuer already carries, applied by + `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`; (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index baedb1d..b201f37 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -67,8 +67,10 @@ Both are deliberate, and both fail **visibly** rather than silently: `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate and the digest were never implicated. The fix is the pin this repository already uses for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now - in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it - to reach the nodes.** + in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the + inventory, never typed on a node. Apply it scoped to that one task: + `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod + --tags hosts`. - **The pull is still unauthenticated** once the node reaches the right endpoint. An unauthenticated client gets nothing from this Gitea's registry, and there is no `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 3a4c759..7f30319 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely. |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | -| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |