Why doesn't Grok recommend my site?
Grok leans heavily on real-time social signals and live web fetches, which makes it behave differently from other assistants — and rewards different things.
Grok draws unusually heavily on X and on live web fetches rather than a slow-moving index. Being technically crawlable is necessary but rarely sufficient: if nobody is linking to or discussing your content where it can see, there is little for it to surface.
Grok has a different centre of gravity
Most assistants answer from a search index built in advance. Grok is built around real-time signals — what is being posted and linked right now — plus live fetches when a URL is in play. That produces a recognisable pattern:
- Recent content is favoured far more strongly than elsewhere. A definitive article from three years ago competes badly with a thread from this morning.
- Content that is being shared and discussed on X has a large advantage, because that is where a great deal of its live signal comes from.
- Direct URL fetches work well. Give Grok your link and it will generally read the page, which is a useful way to isolate whether your problem is access or discovery.
Check the mechanics first anyway
None of the above helps if the page cannot be read. xAI's crawlers identify themselves with tokens including xai-crawler and GrokBot. A catch-all Disallow: / blocks them exactly as it blocks everyone else, and they are frequently missing from the allow-lists people write, because the list was assembled before xAI existed.
User-agent: *
Allow: /
# Named explicitly, so nothing depends on the catch-all
User-agent: xai-crawler
Allow: /
User-agent: GrokBot
Allow: /Discovery is the harder half
If Grok can read your page but never surfaces it unprompted, the gap is that nothing is pointing at it. Practical levers, in rough order of effect:
- Be discussed where it looks. A post on X that links your page, from an account with some reach, does more for Grok visibility than a month of on-page work.
- Publish things worth linking to. Original data, a benchmark, a strong argument. Aggregation and summary get outranked by the thing they summarise.
- Keep it current. A visible, accurate
dateModifiedand genuinely updated content matter more here than for other assistants. - Be specific and quotable. A concrete claim with a number attached travels. A general observation does not.
What not to do
Do not buy engagement, and do not spin up accounts to link your own content. Both are detectable, both are against platform rules, and the failure mode is worse than the problem you started with. The honest version of the same strategy — publishing something people want to share, and being present in the conversation about your subject — is slower and actually compounds.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Whether xAI crawler tokens are allowed or blocked by your robots.txt, alongside every other major AI crawler.
- Whether your content is present in server-rendered HTML rather than requiring JavaScript.
- Whether a machine-readable published and modified date is present, and whether it is plausible.
- Whether Open Graph and Twitter Card tags are correct, which governs how your link renders when shared.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Does Grok respect robots.txt?
xAI publishes crawler documentation and identifies its crawlers by name, which implies compliance. Since the token names are relatively new, the practical risk is not that they ignore your rules — it is that your allow-list predates them and never mentioned them.
Does posting my link on X help?
It is the single most direct lever available for Grok specifically, given how much of its live signal comes from that platform. Posting once from an account with no audience will not move anything; being genuinely discussed will.
Why does Grok summarise my page correctly but never recommend it?
That is the clearest possible signal that access is fine and discovery is the problem. It can read the page when handed the URL, but nothing is surfacing that URL in the first place. Work on links and discussion, not on the page.
Read next
How do I get my site cited by AI assistants?
The mechanics that must be right, then the content decisions that actually earn the citation — in the order worth doing them.
ReadWhy doesn't Perplexity cite my website?
Perplexity cites sources on every answer, which makes it the easiest assistant to diagnose — and the most demanding about page structure.
ReadWhat is generative engine optimization (GEO)?
Optimising to be quoted inside an AI answer rather than ranked in a list of links — what genuinely differs from SEO, and what is the same work under a new name.
ReadWhy does my link preview look wrong on WhatsApp?
Missing Open Graph tags, an image that is too big or the wrong shape, or a cache holding the version from before you fixed it.
Read