How to see GPT and AI crawler traffic in WordPress
“How much of my traffic is bots?” is a question most site owners cannot answer, because the two usual ways of looking both fail at it. Here is why, and how to get a real number.
Why your current analytics cannot tell you
A JavaScript-based tool sees almost none of them. Google Analytics and most hosted analytics count by running a script in a browser. Crawlers generally do not execute JavaScript, so they never appear. Your GA4 numbers are close to crawler-free, which sounds like a feature until you realise it means GA4 cannot answer the question at all.
Server logs see too much. Everything is in there, including uptime monitors, security scanners, preload requests and your own cron. Separating crawlers from people from infrastructure is most of the work, and logs stop at the CDN edge anyway.
Server-side analytics is where it can work, but only if the tool both detects crawlers and reports them rather than silently dropping them. A lot of tools detect and discard, which keeps your human numbers clean and leaves you unable to see the other side.
What you are looking for
The user-agent string, which is how crawlers identify themselves. The named ones are honest about it:
GPTBot- OpenAI, trainingOAI-SearchBot- OpenAI, retrieval for searchClaudeBot- AnthropicPerplexityBot- Perplexity, retrieval, cites sourcesBytespider- ByteDanceCCBot- Common Crawl, widely used as a training sourceGoogle-Extended- a control over training use rather than a distinct crawlerGooglebot,bingbot- search, feeding AI Overviews and Copilot as well as classic results
Matching those by hand is a losing game, because the list changes every few months. A maintained detection library is the practical approach, which is what most tools that do this properly use.
Doing it in Honest Analytics
Crawler activity is a Pro feature and it has its own screen.
Two design decisions matter more than the screen itself:
Detected crawlers are never counted in anything else. Not in pageviews, not in visitors, not in sources, not in the hour-by-day heatmap. Every other number on the site is people. This is the part that makes the human figures trustworthy, and it is worth checking in any tool you use, because “we detect bots” and “we exclude bots from your reports” are different claims.
Crawlers are reported by name and trended. A total is nearly useless. Knowing that one crawler took two thousand pages last Tuesday and had never appeared before is actionable.
What the number usually is
Larger than people expect, and it has grown sharply. Sites that have not looked before commonly find crawlers are a substantial fraction of raw requests, and on smaller sites with a lot of pages they can exceed human traffic outright, because crawler volume scales with how much you have published rather than with how popular you are.
That is the counterintuitive part. A quiet site with two thousand archived posts can be crawled far more heavily than a busy one with fifty pages.
What to do once you can see it
Check your hosting bill and your load. Crawlers consume real resources. A crawler taking your whole archive repeatedly is a performance problem, whatever you think about training data.
Check your overnight traffic. If your day-and-hour pattern shows steady activity at 4am, that is usually crawlers rather than readers, and if your analytics counts them as people your time-of-day pattern is wrong.
Decide about training separately from retrieval. Blocking GPTBot costs you no traffic.
Blocking PerplexityBot removes you from answers that would have cited and linked you. Those are
different decisions and it is worth making them separately rather than blocking everything with
“AI” in the name. How AI crawlers access your site
covers the mechanics.
Watch for new arrivals. A crawler you have never seen appearing at volume is worth a look. The list of what exists changes faster than anyone’s blocklist.
The uncomfortable part
If you have been reporting traffic figures to somebody, and your analytics counts crawlers as people, you have been over-reporting. Possibly by a lot.
That is a good reason to find out which your tool does before the question arrives from someone else. “Every number here is people” should be something you can state and defend, and on most setups it currently is not.