diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md index 623d7d1..45ae653 100644 --- a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -13,9 +13,21 @@ has. `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the -Keycloak issuer pin that is there for the identical reason. **An operator must run the -`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not -something Flux or an agent can do. +Keycloak issuer pin that is there for the identical reason. + +**It reaches the nodes the way every other node setting in this project does — through +Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs, +OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the +*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the +dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no +cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command: + + cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ + -e k3s_hosts=k3s_prod --tags hosts + +`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other +baseline setting. Running the play is an operator action (it has become/root on prod), but +the configuration itself is declared, reviewed and version-controlled — not typed on a node. The docs that described this as an undiagnosed blocker now describe the cause and the fix: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and @@ -86,7 +98,10 @@ with `gitea-settings` the only stage that did not run. ## What I did NOT verify - **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the - nodes is written by the `node_baseline` play, which is an operator action. Until that runs + nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator + action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml` + renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and + `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs and an image exists, "the node now reaches the Gateway" is a design claim. The evidence that it *will* is the Keycloak pin working for the same cause — which I also did not re-measure. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 567ac69..ad87805 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -239,7 +239,8 @@ about runtime behaviour in this document is a design claim, not a measurement. (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same - pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; + pin the Keycloak issuer already carries, applied by + `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`; (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index baedb1d..b201f37 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -67,8 +67,10 @@ Both are deliberate, and both fail **visibly** rather than silently: `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate and the digest were never implicated. The fix is the pin this repository already uses for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now - in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it - to reach the nodes.** + in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the + inventory, never typed on a node. Apply it scoped to that one task: + `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod + --tags hosts`. - **The pull is still unauthenticated** once the node reaches the right endpoint. An unauthenticated client gets nothing from this Gitea's registry, and there is no `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 3a4c759..7f30319 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely. |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | -| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |