Should you block AI crawlers in robots.txt? Search bots vs training bots
Some AI bots fetch pages to answer questions, others collect data to train models. Here is how to tell them apart and what to allow in robots.txt.
Published · Updated
Put this into practice
Also try: Run the full AI visibility check
Why does this matter?
robots.txt is the file where you tell crawlers which parts of your site they may visit. Many sites block everything unfamiliar, including bots that AI assistants use to read pages when someone asks a question. If those bots are blocked, the assistant cannot read your pages directly.
Search bots and training bots are different
AI companies usually run separate bots for separate jobs. Search or answer bots fetch a page so an assistant can use it in a reply. Training bots collect content to build future models. You can allow one group and block the other.
- Answer and search bots: OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot
- Training bots: GPTBot, Google-Extended, Applebot-Extended, CCBot
What should a small business do?
If you want to be found through AI assistants, avoid blocking the answer bots. Whether to allow training bots is a business choice about how your content is reused, and there is no single right answer.
Bot names change over time, so check each company's own documentation before relying on a list. Allowing a bot does not guarantee you will be mentioned. It only means the bot is permitted to read your pages.
How do you check your current rules?
Open yourdomain.com/robots.txt and look for Disallow rules under a User-agent line that matches these bots, or under User-agent: *. A single Disallow: / under User-agent: * blocks every crawler that has no rule of its own.
Frequently asked questions
Does robots.txt stop a bot from reading my page?
It is a request that well-behaved bots follow. It is not a security control, so never rely on it to hide private content.
If I allow AI bots, will I show up in ChatGPT or Perplexity?
Not automatically. Being allowed to crawl is a starting point. Clear, accurate, well-sourced pages matter much more.