Fix robots.txt 404
If robots.txt returns 404, crawlers assume no restrictions. But it's better to have one with Allow: / and Sitemap.
Common causes
- File doesn't exist.
- Wrong path (e.g. /robots.txt in subdir).
- Server returns 404 for /robots.txt.
How to fix
- Create robots.txt at https://yoursite.com/robots.txt.
- Include at least User-agent: * and Allow: /.
- Add Sitemap directive. Verify with AI Bot Access Checker.