Skip to content

ci: notify search engines once a day, from a sitemap snapshot - #103

Merged
brgv-code merged 1 commit into
mainfrom
ci/notify-search-engines-daily
Nov 3, 2026
Merged

brgv-code merged 1 commit into
mainfrom
ci/notify-search-engines-daily

Conversation

@brgv-code

Copy link
Copy Markdown
Owner

Notify ran on every push that touched a page. Three merges to one page in a day sent that URL to IndexNow three times and resubmitted the sitemap to Google three times. IndexNow asks sites to avoid "submitting the same URL many times a day", and Google says "requesting a recrawl multiple times for the same URL won't get it crawled any faster".

Now

  • Runs once a day (06:23 UTC), or by hand with "send every URL".
  • Compares the live sitemap (index and children) with the snapshot from the last successful run, kept in the Actions cache, and sends only URLs that are new or whose lastmod moved. Google gets one sitemap resubmit, only when something changed.
  • No git diff and no wait for the Cloudflare Pages deploy: whatever the sitemap says at run time is live.
  • The snapshot only advances after IndexNow accepted the URLs, so a failed or skipped send is retried the next day.
  • Same environment (search-engines, main only), same secrets, Google and IndexNow functions unchanged. Cache actions pinned by commit like the others.

Honest lastmod (from #96) matters more now: the comparison is only as good as the dates.

Checked: new tests for reading the sitemap, and for finding new or changed URLs (5 passing; web suite 61). A run against the live sitemap without keys reads all 32 URLs, skips both sends with notices, and writes no snapshot. Biome and typecheck clean.

After merging: the first daily run has no snapshot and sends all 32 URLs once. bhargav.dev uses the same pattern (brgv-code/bhargav.dev#73).

On every push, three merges touching one page sent that URL to IndexNow
three times and resubmitted the sitemap three times. IndexNow asks sites
not to resend a URL many times a day; Google says repeat requests do not
get anything crawled faster.

- Daily schedule (and by hand) instead of push. The run compares the
  live sitemap with the last run's snapshot in the Actions cache and
  sends only new URLs and URLs whose lastmod moved.
- No git diff and no wait for the deploy: what the sitemap says is live.
- The snapshot only advances after IndexNow accepted the URLs, so a
  failed or skipped send is retried the next day.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Deploying jlog-web with  Cloudflare Pages  Cloudflare Pages

Latest commit: ad0aca5
Status: ✅  Deploy successful!
Preview URL: https://384c9560.jlog-web.pages.dev
Branch Preview URL: https://ci-notify-search-engines-dai.jlog-web.pages.dev

View logs

@brgv-code
brgv-code merged commit 2d6690a into main Nov 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant