AI Crawler Checker
Can ChatGPT, Claude, Perplexity and Gemini read your website? Check which AI bots your robots.txt lets in.
Search crawlers vs training crawlers
AI companies use different bots for different jobs. Search and user bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) decide whether assistants can find and cite you today. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages for future models. You can allow search and block training if you prefer.
Allow AI search, block training (example)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourproduct.com/sitemap.xmlNext: add an llms.txt and clear answers on your pages. Read how to get recommended by ChatGPT and AEO vs SEO vs GEO.
Frequently asked questions
Can ChatGPT find my website?
Only if OpenAI's search crawler (OAI-SearchBot) and user fetcher (ChatGPT-User) are allowed in your robots.txt, and your pages load without a login. This checker shows both.
Should I block GPTBot or ClaudeBot?
Those collect pages for model training. Blocking them is your choice and does not remove you from ChatGPT or Claude search, which use separate crawlers. Most SaaS marketing sites allow them.
Does Google-Extended affect my Google ranking?
No. Google says Google-Extended controls use of your content for Gemini, and blocking it does not change Google Search. Googlebot handles Search and AI Overviews.
What is llms.txt?
A proposed plain Markdown file at /llms.txt that summarises your site for language models. Support varies, but it is cheap to add. This checker tells you if you have one.
Do all AI crawlers respect robots.txt?
Major companies say their crawlers do, but robots.txt is a request, not a lock. Use authentication for anything private.