There is a new way for a website to fail, and most owners who have it do not know. The site looks fine, ranks reasonably on Google, and yet AI assistants never mention the business, never quote it, never send it a visitor. When I dig into a case like this, the cause is often not the content and not the code. It is a security setting the owner never chose, quietly turning away every AI system that comes to read the site.
Here is what happened. When AI crawlers took off, the companies that sit between websites and the internet reacted. In 2025, Cloudflare, which handles traffic for a large share of the web, started blocking AI crawlers by default for new sites. Hosting companies and security plugins followed with their own “block AI bots” switches, some of them turned on out of the box. The intention was to protect publishers who did not want their writing used for training. The side effect is that a lot of ordinary business sites are now invisible to the exact systems their owners want to be recommended by.
Who is actually knocking
“AI crawlers” is not one thing, and this is the part almost nobody explains. OpenAI alone runs several: GPTBot collects pages for training, OAI-SearchBot builds the index behind ChatGPT’s web search, and ChatGPT-User fetches a page live when a user asks about it. Anthropic has the same split for Claude. Perplexity crawls with PerplexityBot. Google-Extended controls whether Google’s AI models learn from your site, separately from normal Google search. And Bingbot, the ordinary Bing crawler, matters more than ever because Bing’s index feeds Microsoft Copilot and parts of other assistants’ web results.
The distinction that matters for a business: blocking the training crawlers is a philosophical choice, and for some publishers a reasonable one. Blocking the search and live-fetch crawlers is different. That is the machinery assistants use at the moment they answer “who should I hire near me”. Block it and you are simply absent from the conversation, the same way a business with no listed phone number is absent from phone calls.
For a service business that wants customers, I struggle to see the trade. You are not a newspaper protecting a paywall. Your pages exist so that people, and now machines, can find out what you do and pass it along. Being findable by assistants is the whole point of GEO.
The five-minute check
First, open yoursite.com/robots.txt in a browser. It is a public file, anyone can look. Search the text for the names above. If you see GPTBot or ClaudeBot next to the word “Disallow”, your site is telling those systems to leave, whether you meant to or not.
Second, and this is the step people miss, robots.txt only shows what your site asks. Firewalls enforce their own rules before your site is even reached. If you use Cloudflare or a similar service, log in and look for the bot or AI crawler settings; if your site runs on a platform or a security plugin, look for the same toggle there. This is where the silent blocking usually lives.
Third, check the result from the outside. Ask an assistant with live browsing to open your page and summarize it. If it reports that it cannot access the site, you have your answer, and none of your content and structure work is reaching the machines it was written for.
What I do about it
Every site I build ships with a deliberate decision here, in writing: the AI crawlers that matter are explicitly welcomed, this very site included, and after launch I verify from the outside that they really get through. It is one of the first things I look at in the free audit, right before checking what the machines find once they are in, because an invisible site makes every other question moot.
If you have never once thought about this setting, do not feel bad. That is exactly the problem: it was decided for you. The fix is usually one toggle, and knowing it exists is most of the work.