Skip to content
KoishiAI
ไทย

Can AI crawlers read your site?

Paste your robots.txt and see which ChatGPT, Claude, Gemini, Perplexity and other bots are allowed or blocked — both the ones that collect training data and the ones that put your site in AI answers.

Your robots.txt is never sent anywhere — checked in the browser Crawler list checked against official docs on 2026-09-23

Open https://your-site/robots.txt and paste all of it here.

Try:

Summary

Training crawlers blocked
–
AI-search crawlers allowed
–
User-triggered fetchers allowed
–

    Collect data for training

    Blocked: your new content stays out of the next models’ training data.

    Crawler Result Decided by Obeys robots.txt
    GPTBot OpenAI – Yes
    ClaudeBot Anthropic – Yes
    Google-Extended Google · Not a separate crawler: controls whether what Googlebot fetched may train and ground Gemini. No effect on Google Search. – Yes
    Applebot-Extended Apple · Does not crawl: controls whether what Applebot fetched may train Apple’s models. – Yes
    Meta-ExternalAgent Meta – Yes
    CCBot Common Crawl · An open dataset many AI companies train on. – Yes
    Amazonbot Amazon · Amazon says it may be used to train AI models. – Yes
    MistralAI-Training Mistral – Yes

    Search and cite in AI answers

    Blocked: your site won’t be cited by ChatGPT search, Claude, Perplexity and others.

    Crawler Result Decided by Obeys robots.txt
    OAI-SearchBot OpenAI – Yes
    Claude-SearchBot Anthropic – Yes
    PerplexityBot Perplexity – Yes
    Meta-WebIndexer Meta – Yes
    Amzn-SearchBot Amazon – Yes
    MistralAI-Index Mistral – Yes

    Fetch a page when a user asks

    Only visits when someone asks the assistant to read that page. Some say they do not follow robots.txt.

    Crawler Result Decided by Obeys robots.txt
    ChatGPT-User OpenAI · OpenAI says robots.txt rules may not apply, because a user asked for the page. – May not
    Claude-User Anthropic – Yes
    Perplexity-User Perplexity · Perplexity says it generally ignores robots.txt. – May not
    Meta-ExternalFetcher Meta · Meta says it may bypass robots.txt because a user asked. – May not
    Amzn-User Amazon · Amazon says it may not follow every robots.txt directive. – Partly
    MistralAI-User Mistral – Yes

    Classic search engines

    Blocked: you disappear from that search engine entirely.

    Crawler Result Decided by Obeys robots.txt
    Googlebot Google · Blocking it removes you from Google Search, AI Overviews included. – Yes
    Applebot Apple · Powers Siri, Spotlight and Safari. – Yes

    Other

    Crawler Result Decided by Obeys robots.txt
    OAI-AdsBot OpenAI · Checks the safety of pages submitted as ChatGPT ads. – Yes

    Generate robots.txt rules for AI crawlers

    Choose what you allow, then append the result to your existing robots.txt.

      
     
    This generator never blocks Googlebot or Applebot — that would remove your site from search altogether.
     
     
     
    Three kinds of AI crawler — decide on each separately
     
       
    • Training crawlers (GPTBot, ClaudeBot, CCBot…) collect content for datasets. Blocking them does not affect whether people can find you.
    • AI-search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot…) get your site cited in answers. Blocking them costs you traffic from AI.
    • User-triggered fetchers (ChatGPT-User, Claude-User…) only come when someone asks the assistant to read your link. OpenAI, Perplexity and Meta say these may not follow robots.txt.
    • “No training, but do find me” is possible: every major operator now uses a different crawler name for each purpose.
      •  
     
    Worth knowing
     
       
    • Google-Extended is not a crawler that visits. It tells Google whether what Googlebot fetched may train and ground Gemini. Google says it has no effect on Google Search ranking.
    • Applebot-Extended likewise does not crawl; it controls whether what Applebot fetched may train Apple’s models.
    • Names must match exactly: an “Applebot” group does not cover “Applebot-Extended”. With no group of its own, a crawler follows the * group (RFC 9309).
    • ByteDance’s Bytespider has no reachable official documentation and is widely reported to ignore robots.txt, so it is not in the table.
    • robots.txt is a request, not enforcement. To really keep a crawler out, block it at your server or CDN.
    • If your server answers robots.txt with a 5xx error, crawlers must treat the whole site as disallowed. A 4xx (such as 404) means everything is allowed.
      •  
     
    How it checks
     
       
    • Decisions use the robots-parser library, which follows RFC 9309: case-insensitive group selection by crawler name, the longest matching rule wins, Allow wins a tie, and * and $ are supported.
    • We compared this tool with the Protego library on the real robots.txt files of 31 Thai and international sites: 64,768 decisions, 98 differences. 74 were Applebot-Extended, where Protego matches names by substring against RFC 9309; 24 were paths ending in a bare “?”, which this tool drops.
    • On 4,000 random, structurally odd files (96,000 decisions) there were 338 differences, all from a Crawl-delay line between user-agent lines, which this tool skips exactly as Google’s reference parser does.
    • robots-parser on its own departs from RFC 9309 in three places, all corrected in our wrapper: /robots.txt is always readable, a crawler’s own group with no rules allows everything, and a Sitemap or Crawl-delay line does not split a group.
    • Crawler names, purposes and robots.txt behaviour come from each operator’s documentation, last checked 2026-09-23.
      •  
     
    FAQ
     
     
     If I block GPTBot, will I disappear from ChatGPT? 
    Not from search. GPTBot only collects training data; showing your site in ChatGPT search uses OAI-SearchBot, which is separate.
     
     Will blocking Google-Extended hurt my Google ranking? 
    Google says it does not affect inclusion in Google Search and is not a ranking signal.
     
     Is my robots.txt sent anywhere? 
    No. Everything runs in your browser. This page does not fetch your robots.txt itself — you paste it in.
     
     Why isn’t crawler X listed? 
    Only crawlers with official documentation stating their name and purpose are included: 23 today, 8 of them for training.
     
     
     
    Sources
     
       
    • OpenAI https://developers.openai.com/api/docs/bots
    • Anthropic https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
    • Google https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
    • Apple https://support.apple.com/en-us/119829
    • Perplexity https://docs.perplexity.ai/guides/bots
    • Meta https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
    • Common Crawl https://commoncrawl.org/ccbot
    • Amazon https://developer.amazon.com/amazonbot
    • Mistral https://docs.mistral.ai/robots
      •  
    • RFC 9309 — Robots Exclusion Protocol
      •