How do I check if AI crawlers can read my site?
Four tests you can run yourself in about ten minutes, in the order that finds the problem fastest.
Fetch your robots.txt and read every group; fetch your page with curl and check your content is in the HTML; look for noindex directives; then ask an assistant to summarise your URL. Those four tests find nearly every cause.
Test one: read the robots.txt that is actually served
curl -s https://yoursite.com/robots.txtRead all of it. Three things to look for:
- Any
Disallow: /, particularly underUser-agent: *. That blocks every crawler without its own more specific group. - Whether the AI crawlers you care about are named. The most specific matching group wins, so a named group is the only reliable way to allow something.
- Whether this file is yours. Some hosts and CDNs serve their own at the edge regardless of what you uploaded, which is a genuinely common and genuinely maddening cause.
Test two: fetch the page without a browser
curl -sL https://yoursite.com | wc -c
curl -sL https://yoursite.com | grep -i "a distinctive sentence from your page"The first command tells you how much HTML comes back. A few kilobytes from a page that looks full in a browser means the content is added by JavaScript. The second confirms directly: if your text is not in the response, no crawler that does not render JavaScript can see it.
Test three: look for directives that remove you independently
robots.txt is not the only way to disappear. Check for both of these:
# In the HTML
<meta name="robots" content="noindex">
# In the response headers
curl -sI https://yoursite.com | grep -i x-robots-tagEither one removes you from indexes regardless of how permissive your robots.txt is. A noindex left behind after a staging deployment is one of the most common single-line causes of a site vanishing.
Test four: ask an assistant directly
Paste your URL into ChatGPT, Claude or Perplexity and ask what the page says. This tests the live-fetch path end to end, including anything your CDN is doing that the earlier tests missed.
- Accurate summary: access is fine. Any visibility problem is about content and discovery, not mechanics.
- "I can't access that": the live-fetch crawler is blocked — by robots.txt, by bot protection, or by a firewall rule.
- Summary of the wrong thing: it read a challenge page, an error page, or a shell without your content.
The cause people miss
Bot protection. Cloudflare, and similar services, can serve a challenge to anything they classify as automated. Your browser passes it invisibly; crawlers cannot pass it at all. Everything looks perfect while the site is unreadable to every machine consumer.
The tell is a mismatch: curl returns a challenge page or a 403 while the browser loads normally. The fix is in your CDN dashboard — allow the verified AI crawler user-agents — not on your site.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Twelve named AI and search crawlers, each with an allowed or blocked verdict and the rule responsible.
- The proportion of page text present without JavaScript.
- Whether a noindex meta tag or X-Robots-Tag header is present.
- Whether the response resembles a bot-protection challenge rather than the page itself.
- Whether the robots.txt served differs from what a normal browser request receives.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Is there a tool that checks all of this at once?
That is precisely what the audit on this site does — it reports every named crawler with the exact rule that governs it, measures how much content survives without JavaScript, and flags noindex directives and challenge pages. The manual tests above are worth knowing anyway, because they tell you what is being measured.
Can I test with a user-agent switcher?
Partially. Sending User-agent: GPTBot with curl shows you whether your server or CDN treats that agent differently. It does not tell you whether the crawler would have chosen to fetch you, and it does not simulate robots.txt compliance, since curl ignores the file.
My site is fine on Google but invisible to AI. How?
Almost always JavaScript. Googlebot renders it; AI crawlers largely do not. Google seeing your content is not evidence that anything else can.
Read next
Does JavaScript hurt AI crawlers?
Yes, more than it hurts search engines. If your content only exists after JavaScript runs, most AI crawlers see an empty page.
ReadWhy is my website not showing on ChatGPT?
The four things that decide whether ChatGPT can find, read and cite your site — and how to tell which one is stopping you.
ReadHow do I fix robots.txt blocking Google?
How to read the file correctly, what the precedence rules actually are, and the three mistakes that cause almost every accidental block.
ReadWhy can't Claude see my website?
Anthropic runs three separate crawlers with three separate purposes. Blocking the wrong one is the usual reason a site is invisible.
Read