Fix Wildcard Blocking AI Crawlers
Disallow: /* or broad path patterns can block more than intended. AI crawlers may be blocked from key sections.
Common causes
- Disallow: /* or Disallow: *.
- Disallow: /.* or regex-style pattern.
- Wildcard in path blocks whole site.
How to fix
- Use specific paths instead of wildcards.
- Test with User-Agent Rules for AI.
- Allow before Disallow for paths you want open.