How AI crawlers access your WordPress website

Your site is no longer read mostly by people and search engines. A third category has arrived, it is growing quickly, and most site owners have no idea how much of their traffic it accounts for.

This is what is out there, how it reaches you, and what your options are. Seeing it in your own numbers is a separate post.

The crawlers, and what they are for

They are not all doing the same job, and the distinction matters when you decide what to allow.

Training crawlers collect text to train models. OpenAI’s GPTBot, Anthropic’s ClaudeBot, Bytespider from ByteDance, CCBot from Common Crawl. Content taken now may influence a model released much later, and there is no mechanism to withdraw it afterwards.

Retrieval crawlers fetch pages to answer a question being asked right now, and usually cite the source. PerplexityBot, OAI-SearchBot, and the fetching behaviour behind various assistants. These can send you traffic.

Search crawlers with an AI dimension. Googlebot feeds both classic search results and AI Overviews, and Google-Extended is a separate control for training rather than a separate crawler. Bing’s crawler feeds Copilot the same way.

The practical consequence: blocking everything with “AI” in the name can also remove you from answer engines that would have cited and linked you. Those are different trades and worth deciding separately.

How they find you

The same way anything does, mostly. Links from other sites, your sitemap, your RSS feed, and pages they have crawled before and return to.

A few WordPress-specific things make it easier than people realise:

• Your sitemap is public. WordPress generates one by default at /wp-sitemap.xml, and it is a complete list of everything you have published.
• Your RSS feed is a clean full-text source on a lot of themes, which is more convenient to ingest than parsing HTML.
• Archive, category, tag and author pages multiply the routes to the same content.
• The REST API exposes post content as JSON at /wp-json/wp/v2/posts unless something has turned it off.

None of that is a vulnerability. It is WordPress working as designed. It does mean “I only linked it from one place” is not much of a defence.

What robots.txt actually does

robots.txt is a request. Well-behaved crawlers honour it, and the major named ones generally do, because being caught ignoring it is a reputational problem for them. Crawlers that do not care will ignore it, and some disguise themselves as browsers, in which case nothing in a text file will help.

A rule blocking a training crawler looks like this:

User-agent: GPTBot
Disallow: /

Points worth knowing before you write any:

• It is per user-agent, so a list needs maintaining as new ones appear.
• It does not remove anything already taken.
• Blocking a retrieval crawler removes you from answers it would have cited, links included.
• Google-Extended controls training use without affecting your search ranking. They are deliberately separable.

If you want to signal permissions more explicitly there is a growing convention of publishing an llms.txt, though support is inconsistent and it is a statement of intent rather than an enforcement mechanism.

Should you block them?

There is no general answer, but the question decomposes usefully.

Blocking training crawlers costs you nothing in traffic, because they were never going to send any. If you object to your writing being used as training data, this is the lever, and it is cheap.

Blocking retrieval crawlers does cost you. They are increasingly how people find things, and they cite sources. Blocking them is closer to blocking a search engine than to protecting your content.

Blocking search crawlers is almost never right, even where they feed AI Overviews.

For most sites the reasonable position is: allow search and retrieval, decide about training on principle, and above all find out what the actual volumes are before doing anything. Which is the part almost nobody does.

Start by measuring

It is difficult to make a sensible decision about crawler traffic while your analytics is either ignoring crawlers entirely or quietly counting them as people. Both are common, and the second is worse, because it inflates your numbers in a way that looks like an audience.

Honest Analytics never counts detected crawlers in pageviews, visitors or anything else, and Pro reports them by name on their own screen. That gives you the actual figure to decide from rather than an impression.