Why can't Claude see my website?
Anthropic runs three separate crawlers with three separate purposes. Blocking the wrong one is the usual reason a site is invisible.
Check whether ClaudeBot, Claude-SearchBot or Claude-User is disallowed in your robots.txt — including by a catch-all User-agent: * rule. If all three are allowed and Claude still cannot read the page, your content almost certainly requires JavaScript to appear.
Three crawlers, three jobs
| Crawler | Purpose | Effect of blocking |
|---|---|---|
ClaudeBot | Gathers content for model training | Excluded from training corpora. No effect on live use. |
Claude-SearchBot | Indexes pages so Claude can retrieve and cite them | Claude cannot surface your site when searching. |
Claude-User | Fetches a page in the moment, because a user asked about it | A user who gives Claude your URL gets an error instead of your content. |
The last one is the most costly to block and the least understood. Claude-User is not a crawler harvesting your site in bulk — it is one request, made because a person explicitly asked about your page. Blocking it means the interested visitor gets nothing.
The catch-all trap
This is the most common configuration mistake in this whole area:
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /This says: Google may read everything, nobody else may read anything. Every Claude crawler is blocked, along with every OpenAI and Perplexity crawler, without any of them being mentioned. Sites arrive at this state by pasting a snippet that was written for a staging server.
If you want to allow AI crawlers, name them explicitly. The most specific matching user-agent group wins, and a named group overrides the catch-all entirely — including any rules the catch-all had that you meant to keep, so repeat what you need:
User-agent: *
Disallow: /admin/
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
User-agent: Claude-User
Allow: /
Disallow: /admin/Verify rather than assume
- Open
yoursite.com/robots.txtand read every group, not just the first. - Confirm the file being served is the file you wrote — some hosts and CDNs substitute their own.
- Fetch your page with
curl -sLand check your real content is in the response body. - Look for a
noindexmeta tag orX-Robots-Tagheader, which are separate from robots.txt and independently sufficient to remove you. - Check for a bot-protection layer. Cloudflare and similar services can serve a challenge page to anything they consider automated, which crawlers cannot pass.
curl request returns a challenge page or a 403 while the browser loads fine, that is the problem — and it is fixed in your CDN dashboard, not on your site.When everything is allowed and it still does not work
Two remaining possibilities. Either the content is not in the HTML — see the JavaScript guide below — or the page is genuinely reachable and Claude is choosing other sources, which is a content question rather than a technical one.
You can distinguish the two in seconds. Paste your URL into Claude and ask it to summarise the page. If it reports it cannot access the content, the problem is mechanical. If it summarises the page accurately but does not cite you elsewhere, the mechanics are fine and the issue is that your page is not the strongest answer to the question being asked.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Each of
ClaudeBot,Claude-SearchBotandClaude-Userreported separately, with the exact robots.txt rule that applies to each. - Whether a
noindexdirective is present in the HTML or in the response headers. - Whether the response looks like a bot-protection challenge rather than your page.
- How much of the page text survives without executing JavaScript.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Is Claude-User the same as a crawler?
No, and the distinction matters. Claude-User fetches a single page because a user asked about it in that moment. There is no bulk crawling and no training use. Blocking it does not protect your content from anything — it just breaks the experience for someone who was actively interested in your page.
Can I allow Claude to read my site but not train on it?
Yes. Allow Claude-SearchBot and Claude-User, disallow ClaudeBot. That keeps you retrievable and citable while excluding your content from training collection.
Does Claude respect robots.txt?
Anthropic publishes that its crawlers do, and they identify themselves honestly in their user-agent rather than impersonating a browser. That also means the reverse holds: if you block them, you really are removed.
Read next
Why is my website not showing on ChatGPT?
The four things that decide whether ChatGPT can find, read and cite your site — and how to tell which one is stopping you.
ReadShould I block AI crawlers from my site?
An honest look at what you gain and what you give up, and why the answer is usually "block some, allow others".
ReadDoes JavaScript hurt AI crawlers?
Yes, more than it hurts search engines. If your content only exists after JavaScript runs, most AI crawlers see an empty page.
ReadHow do I check if AI crawlers can read my site?
Four tests you can run yourself in about ten minutes, in the order that finds the problem fastest.
Read