docs(node): name the exact Ansible command that applies the pin, not 'run the play'
Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.
The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.
Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
523a463fd7
commit
90a79d29d3
4 files changed
+26
-8
No files matched your search
@@ -13,9 +13,21 @@ has.
|
||||
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
|
||||
`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
|
||||
Keycloak issuer pin that is there for the identical reason. **An operator must run the
|
||||
`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not
|
||||
something Flux or an agent can do.
|
||||
Keycloak issuer pin that is there for the identical reason.
|
||||
|
||||
**It reaches the nodes the way every other node setting in this project does — through
|
||||
Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
|
||||
OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
|
||||
*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
|
||||
dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
|
||||
cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
|
||||
|
||||
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
|
||||
-e k3s_hosts=k3s_prod --tags hosts
|
||||
|
||||
`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
|
||||
baseline setting. Running the play is an operator action (it has become/root on prod), but
|
||||
the configuration itself is declared, reviewed and version-controlled — not typed on a node.
|
||||
|
||||
The docs that described this as an undiagnosed blocker now describe the cause and the fix:
|
||||
`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
|
||||
@@ -86,7 +98,10 @@ with `gitea-settings` the only stage that did not run.
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
|
||||
nodes is written by the `node_baseline` play, which is an operator action. Until that runs
|
||||
nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
|
||||
action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
|
||||
renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
|
||||
`state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
|
||||
and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
|
||||
that it *will* is the Keycloak pin working for the same cause — which I also did not
|
||||
re-measure.
|
||||
|
||||
@@ -239,7 +239,8 @@ about runtime behaviour in this document is a design claim, not a measurement.
|
||||
(i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
|
||||
router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
|
||||
`gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
|
||||
pin the Keycloak issuer already carries; an operator must run the `node_baseline` play;
|
||||
pin the Keycloak issuer already carries, applied by
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
|
||||
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
|
||||
the credential decision now being local Gitea accounts with PATs, that means a
|
||||
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
|
||||
|
||||
@@ -67,8 +67,10 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
|
||||
and the digest were never implicated. The fix is the pin this repository already uses
|
||||
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
|
||||
in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it
|
||||
to reach the nodes.**
|
||||
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
|
||||
inventory, never typed on a node. Apply it scoped to that one task:
|
||||
`cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
||||
--tags hosts`.
|
||||
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
|
||||
unauthenticated client gets nothing from this Gitea's registry, and there is no
|
||||
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
|
||||
|
||||
@@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely.
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
|
||||
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
|
||||
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
|
||||
|
||||
Reference in new issue
Block a user