Fix Blocked AI Crawlers
If your robots.txt blocks AI crawlers (GPTBot, ClaudeBot, PerplexityBot), your content won't appear in ChatGPT, Claude, or Perplexity answers. AI systems rely on crawling—blocking them means zero AI visibility.
Common causes
- robots.txt has Disallow: / for AI bot user-agents.
- Critical paths (e.g. /blog, /docs) are disallowed.
- No explicit Allow rule for AI crawlers.
How to fix
- Add Allow rules for AI bots before Disallow if you want partial access.
- Use our Robots.txt for AI Bots tool to see all rules that apply to AI crawlers.
- Test with AI Bot Access Checker before changing production.