Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
fda5c31a94 |
No files matched your search
@@ -74,26 +74,12 @@ name: renovate
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
|
||||
# `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
|
||||
# it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
|
||||
# merely near it.
|
||||
#
|
||||
# WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
|
||||
# run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
|
||||
# 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
|
||||
# Every nightly run would have started, logged "not within schedule", opened zero PRs and
|
||||
# exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
|
||||
# green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
|
||||
# needed updating".
|
||||
#
|
||||
# THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
|
||||
# EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
|
||||
# EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
|
||||
# 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
|
||||
# schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
|
||||
# early-morning-UTC value without redoing this arithmetic.
|
||||
- cron: "15 8 * * *"
|
||||
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
|
||||
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
|
||||
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
|
||||
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
|
||||
# of the repo window means a run is always in flight when the window opens.
|
||||
- cron: "15 3 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
|
||||
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+32
-1
@@ -143,8 +143,39 @@ env:
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
#
|
||||
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
#
|
||||
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
|
||||
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
|
||||
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
|
||||
# single lookup, which is maximum exposure to any momentary gap; while
|
||||
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
|
||||
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
|
||||
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
|
||||
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
|
||||
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
|
||||
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
|
||||
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
|
||||
#
|
||||
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
|
||||
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
|
||||
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
|
||||
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
|
||||
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
|
||||
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/pihole-local-ttl
|
||||
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
|
||||
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
|
||||
**asymmetry in cacheability** that we created.
|
||||
|
||||
Measured on prod 2026-10-18:
|
||||
|
||||
```
|
||||
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
```
|
||||
|
||||
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
|
||||
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
|
||||
rather than forwarding live. So:
|
||||
|
||||
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
|
||||
lookup, which is maximum exposure to any momentary gap;
|
||||
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
|
||||
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
|
||||
address and serves it to **every LAN client** for five minutes.
|
||||
|
||||
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
|
||||
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
|
||||
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
|
||||
|
||||
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
|
||||
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
|
||||
regenerated with the canonical render command.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
|
||||
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
|
||||
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
|
||||
143 (decrementing), which a pure forwarder could not do.
|
||||
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
|
||||
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
|
||||
address), so a fallback always has a wrong answer available.
|
||||
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
|
||||
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
|
||||
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
|
||||
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
|
||||
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
|
||||
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
|
||||
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
|
||||
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
|
||||
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
|
||||
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
|
||||
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
|
||||
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
|
||||
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
|
||||
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
|
||||
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
|
||||
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
|
||||
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
|
||||
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
|
||||
public answer latches again. That is a real availability problem, not a caching one.
|
||||
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
|
||||
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
|
||||
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
|
||||
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
|
||||
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
|
||||
that decision, and it is deliberately **not** in this change.
|
||||
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
|
||||
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
|
||||
there. Only this one workstation's resolver chain was inspected.
|
||||
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
|
||||
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
|
||||
next recurrence would again be found by a human noticing. That gap is not closed here.
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/renovate-cron-window
|
||||
title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
|
||||
never have produced a single PR.**
|
||||
|
||||
Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
|
||||
with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
|
||||
re-checks it and declines to open PRs outside it. So:
|
||||
|
||||
```
|
||||
cron 03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
|
||||
window -> 03:00-06:00 local
|
||||
inside? -> NO (3h45m before the window opens)
|
||||
```
|
||||
|
||||
The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
|
||||
in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
|
||||
have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
|
||||
green tick forever, indistinguishable from "nothing needed updating". That is the same
|
||||
"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
|
||||
pending updates PR #73 was written to surface.
|
||||
|
||||
Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
|
||||
position moves:
|
||||
|
||||
| | local window | in UTC |
|
||||
|---|---|---|
|
||||
| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
|
||||
| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
|
||||
|
||||
`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
|
||||
A cron cannot follow DST, so this is the only way to get it right once.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
|
||||
`America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
|
||||
* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
|
||||
both inside `03:00–06:00`. Computed, not reasoned.
|
||||
* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
|
||||
`timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
|
||||
* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
|
||||
alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
|
||||
when PR #73 was authored.
|
||||
* The edited workflow parses as YAML.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
|
||||
has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
|
||||
usable PR.
|
||||
* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
|
||||
remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
|
||||
`customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
|
||||
Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
|
||||
the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
|
||||
including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
|
||||
change it**, because guessing at a config key that could fail validation and take the whole run
|
||||
down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
|
||||
dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
|
||||
validation warning naming `regex`.
|
||||
* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
|
||||
even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
|
||||
schedule". That makes the first dispatch's output less informative than it looks; to see what
|
||||
Renovate *would* do, either dispatch inside the local window or override the schedule for that
|
||||
run. Not tested either way.
|
||||
* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
|
||||
Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
|
||||
produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
|
||||
does.
|
||||
* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
|
||||
return. A static check comparing the two is feasible and is not written.
|
||||
Reference in new issue
Block a user