Skip to content
Approvalens

Free tool · no sign-up

Can ChatGPT see your site?

We check whether GPTBot, ClaudeBot, PerplexityBot and 11 other AI crawlers may read your homepage: the exact robots.txt rule for each one, what your server answers when they knock, noai tags, JavaScript, llms.txt and your sitemap. Free, no sign-up, about 20 seconds.

Check a live site

Free · no sign-up · homepage only · about 20 seconds

Free homepage check. Every full Site report runs it on every page — scan the whole site.

What we test

Six things decide whether an AI assistant can read your site

  1. robots.txt, per crawler

    We read your robots.txt the way RFC 9309 describes and find the group and the exact rule that applies to each AI crawler. Exact.

  2. Server access, including Cloudflare

    We request your homepage with each crawler's user agent and spot refusals, challenge pages and rate limits. Simulated.

  3. noai directives

    Robots meta tags and X-Robots-Tag headers with noai, noimageai, noindex or nosnippet.

  4. JavaScript dependency

    We compare the words in the raw HTML with the words after the page renders. Most AI crawlers don't run JavaScript.

  5. llms.txt

    Whether you have one and, if so, whether it follows the proposed format, problem by problem.

  6. Sitemap

    We sample pages from your sitemap and check which AI crawlers robots.txt keeps out of them.

Who we test

14 AI crawlers and their robots.txt names

Each links to its vendor's own documentation.

Why AI crawlers are different from Googlebot

Each AI company runs several crawlers, and they do different jobs. OpenAI's GPTBot collects pages to train models, OAI-SearchBot finds pages to show in ChatGPT search, and ChatGPT-User fetches a page when someone asks ChatGPT to. Anthropic and Perplexity split their crawlers the same way. Blocking one does not block the others.

That means a single line in robots.txt can keep your site out of AI training and still let assistants cite you, or, by accident, do the opposite. The check shows the verdict for each crawler, so you can see which of the two you have.

How to read the result

  • Allowed: robots.txt lets the crawler fetch your homepage and our simulated request got the page.
  • Blocked by robots.txt: a rule in your file tells the crawler not to fetch the homepage. We quote the rule, its line and its group. This result is exact.
  • Blocked at server: robots.txt says yes, but your server or firewall refused our request or showed a challenge page. This result is simulated; confirm it in your firewall log.
  • Unclear: the request was rate-limited, failed or returned a much shorter page than a browser gets.

Blocking training is a choice, blocking search is usually a mistake

Many sites block training crawlers on purpose, and that is a legitimate decision. We report it as information, not as a problem. Search and assistant crawlers are different: if they are blocked, your pages can't appear with a link in ChatGPT, Claude or Perplexity answers.

None of this affects Google Search or AdSense. Google-Extended, for example, only controls whether Google may use your pages for Gemini; Google says it does not affect your inclusion or ranking in Search.

Questions

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. OpenAI documents three agents: GPTBot collects content for training, OAI-SearchBot finds pages to show in ChatGPT search results, and ChatGPT-User fetches a page when a user asks. Blocking GPTBot keeps your pages out of training; to appear in ChatGPT search, OAI-SearchBot must be allowed.

Source:developers.openai.com

What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?

Anthropic says ClaudeBot collects public web content that may be used to train its models, Claude-SearchBot crawls to improve the quality of search results, and Claude-User fetches pages when a Claude user asks a question. Anthropic states that its bots honour robots.txt, and each one can be blocked separately.

Source:support.claude.com

Does Perplexity follow robots.txt?

Perplexity documents two agents. PerplexityBot finds and links sites in Perplexity's search results and follows robots.txt. Perplexity-User fetches a page because a user asked for it, and Perplexity says it generally ignores robots.txt for those requests.

Source:docs.perplexity.ai

Does blocking Google-Extended hurt my Google ranking?

No. Google describes Google-Extended as a standalone robots.txt product token that controls whether your content may be used for its Gemini models; it is not a separate crawler and it does not affect inclusion or ranking in Google Search.

Source:developers.google.com

Why would Cloudflare block AI crawlers on my site?

Cloudflare offers a setting that blocks known AI crawlers across a site, and it can be switched on from the dashboard in one step. If it is on, AI crawlers get refused or challenged even when your robots.txt allows them. Our server test shows this, but only as a simulation, because Cloudflare can recognise real crawlers by their network.

Source:developers.cloudflare.com

What is llms.txt, and do I need one?

llms.txt is a proposal for a Markdown file at /llms.txt that gives AI tools a short map of your site: a title, a summary and links to your key pages. It is not a standard and the major AI crawlers do not say they use it, so we report a missing file as information. If you have one, we check it against the proposed format.

Source:llmstxt.org

Do noai and noimageai meta tags work?

They are not part of any standard, and Google's list of supported robots meta rules does not include them. They state your wish but do not block a crawler. To keep a crawler out, use robots.txt, and to keep pages out of search entirely, use noindex.

Source:developers.google.com

How exact is this check?

The robots.txt part is exact: we read your file the way RFC 9309 describes, with the most specific group and the longest matching rule. The server part is simulated: we send each crawler's user agent from our own servers, and a firewall that verifies bots by IP address may treat the real crawler differently.

Source:rfc-editor.org

Guides

Guides that use this tool

Where this check comes up, and how to fix what it finds.

Part of the Site report

AI visibility is part of every full report

This free check reads your homepage. The full Site report runs the same tests on every page it crawls, with the AdSense readiness review, the SEO audit and a security check alongside: one scan, one report, one price.

  • 01AdSense readiness

    Will AdSense approve it?

    • Google's policies and content-value signals
    • Privacy, about and contact pages
    • ads.txt and the AdSense crawler
    • A readiness score, blockers first

    241 checks behind the score

  • 02SEO audit

    Can search engines crawl and understand it?

    • Indexing, canonicals and hreflang
    • Titles, descriptions and headings
    • Internal links, orphans and click depth
    • Speed, images, mobile and structured data

    107 SEO checks

  • 03AI visibilityYou're here

    Can ChatGPT, Claude and Perplexity read it?

    • robots.txt rules for each AI crawler
    • Server and Cloudflare access (simulated)
    • noai tags and JavaScript-only content
    • llms.txt, checked and built for you

    14 AI crawlers tested

Pricing

Start free. The whole site is one report.

  1. 01Free check

    $0homepage

    • Your homepage against 14 AI crawlers
    • robots.txt rules and simulated server access
    • noai, JavaScript, llms.txt and sitemap
    • A shareable result link
  2. 02Full site report

    $14.99one-time, per site

    • Your whole site crawled, up to 20,000 pages
    • Every page against the 14 AI crawlers we test
    • An llms.txt built from your own pages
    • The AdSense readiness review, the SEO audit and a security check in the same report
    • Every problem with its evidence and fix steps
    • Unlimited rescans and site monitoring for one month

All pricesMonitoring plans

Next step

One check here, the whole site in the report

This is one of 258 checks. Scan the whole site for the full picture; the first 50 pages are free.

First 50 pages free · no sign-up · no card