Tools · Free

AI crawler checker

Find out whether ChatGPT, Claude, Perplexity and Google's AI features are allowed to read your site. The checker reads your robots.txt and tells you which crawlers it lets in, which it blocks, and whether that's what you want.

Search crawlers and training crawlers

AI companies run two kinds of crawler. Search crawlers read your pages so their assistant can quote and link to you when someone asks a question. Block them and you can't be recommended. Training crawlers collect pages to train future models. Blocking those is a business choice, and it doesn't stop you appearing in answers.

Most businesses that want to be found should let the search crawlers in. Whether you let training crawlers in is up to you. The checker groups them so you can see which is which.

What about Google?

Google's AI Overviews and AI Mode use the normal Googlebot. If Googlebot can crawl you, you can appear in them. Google-Extended only controls whether Google uses your content for Gemini, and blocking it doesn't affect your rankings.

What about llms.txt?

The checker tells you whether you have an llms.txt file. Google has said it doesn't use one, and no major AI company has said it relies on it. It does no harm, but it isn't what gets you cited. Clear, specific pages that answer real questions are. I wrote about how one site earned 500,000 AI citations.

How it works

The checker fetches your robots.txt from Cloudflare's network and reads the rules the way the crawlers do: a crawler follows the group that names it, or the * group if none does, and the most specific rule wins. Nothing is stored.

Questions

Does blocking GPTBot stop ChatGPT recommending me?

No. GPTBot collects content for training. ChatGPT search uses OAI-SearchBot, so blocking GPTBot alone keeps you out of training without affecting search answers.

How do I allow AI crawlers in robots.txt?

A crawler is allowed unless a rule blocks it. Remove any Disallow: / line under its name, or under User-agent: * if that group blocks everything.

Does robots.txt stop AI using my content?

Only for crawlers that follow it. The major search and training crawlers say they do. OpenAI says robots.txt may not apply to ChatGPT-User, which opens pages a user asks for.

How long until a change takes effect?

OpenAI says about 24 hours for ChatGPT search. Other crawlers pick up changes the next time they fetch your robots.txt, usually within a day.

Want to be recommended by AI?

Letting the crawlers in is step one. Getting chosen is the rest of the work, and it's what my AI search optimisation service covers. Email me at nelson [@] nelsonisibor.com with your site.