Live demo · web operations · 2026

A crawler log: which bots read this site

Every crawler request that reaches WordPress is counted on this site’s server: search engines, AI crawlers and assistants, link previews and SEO tools. Repeats from the same crawler and address to the same page within a minute are rolled into one row with a hit count. The page shows the last 28 days: which crawlers came, what they asked for and when. Ordinary visitors are never logged.

Year
2026
28
days shown
60
days kept, then deleted

Results

  • A request is written after WordPress has answered, with the status it got.
  • Every request is counted. A repeat from the same crawler and address to the same page within a minute adds a hit to that row, so a bot hammering a page shows its full count without filling the table.
  • The public view drops the last part of every IP address and names only crawlers on the known list.

My role

Designed and built it: the logging, its table, the endpoint and the page.

Stack

  • PHP 8
  • WordPress hooks
  • User-agent matching
  • MySQL table
  • WordPress REST API

This site’s crawlers

Running on this server. Also on Live demos.

Crawlers on this site

loading…

Loading the crawler log…

How a crawler is recognized

A request is logged when its user agent names a known crawler, or isn’t a browser’s and calls itself a bot, crawler or spider, or is a common HTTP library. A phone browser whose model name happens to end in “bot” isn’t logged.

Why it’s worth knowing

AI assistants and search engines now read a site on someone’s behalf. Knowing which ones come, and what they ask for, says more about how a site is found than visitor counts alone.