Is my website blocking AI?
Possibly, and you would not know from looking. A site can load perfectly for a person and still turn away the crawlers that ChatGPT, Claude, and Perplexity use to read it. The block usually comes from a hosting or security default rather than a decision you made, and Google is often allowed through while the AI crawlers are not. The only reliable way to know is to fetch the site the way a machine does and see what comes back.
Two separate systems decide whether a machine can read your site. The first is robots.txt, the file where you state which crawlers are welcome. The second is the security layer in front of the site, a firewall, a bot filter, or a content delivery network, which can refuse a request before your server ever sees it. Most owners only ever check the first. Our study of 252 Northwest Ohio business websites found 6.3 percent (16 of 252) blocking an AI agent at the security layer while their robots.txt welcomed it. Google’s own crawler is usually on the allow list, so search rankings hold and nothing in Search Console changes.
Where this matters
Blocking the crawlers that train models is a legitimate choice and costs little visibility. Blocking the crawlers that fetch and cite is what removes a business from AI answers, and the two are often caught by the same broad rule or the same default. Since mid-2025 new Cloudflare accounts block AI crawlers unless told otherwise, and several hosts ship a bot wall that challenges anything that cannot run JavaScript, which describes every AI fetcher. A person in a browser passes that challenge without seeing it. If your site sits on a managed platform, the switch may live in the host’s control panel or behind a support ticket rather than anywhere you can edit. Check first, then decide which crawlers you actually mean to keep out.
To find out in about a minute, run the AI visibility check; for why the block happens and what to do about it, see how AI describes your business. Fixing what it finds is part of the practice’s SEO process.
Corrected 2026-10-18: this note first gave the figure as 17.9 percent. That count included 26 sites the study’s check never reached; the measured figure is 6.3 percent, 16 of 252. The study carries the full correction note.