Writing a Good robots.txt File: The Complete Guide
Your robots.txt file is a simple text file that tells web crawlers which parts of your site they can and cannot access. Getting it right is important for both SEO and AI visibility. Here's everything you need to know.
What robots.txt actually does
When a crawler visits your site, the first thing it checks is yoursite.com/robots.txt. This file contains rules about which URLs the crawler is allowed to access. It's the bouncer at the door of your website.
Basic syntax
- User-agent: Googlebot
- Allow: /
- User-agent: *
- Disallow: /admin/
- Disallow: /private/
Sitemap: https://yoursite.com/sitemap.xml
Each rule starts with a User-agent (which crawler this applies to), followed by Allow or Disallow directives for specific paths.
What to block
- Admin and login pages (/admin/, /wp-admin/, /login/)
- Internal search result pages (/?s= or /search/)
- Shopping cart and checkout pages
- Staging or development environments
- API endpoints that don't contain public content
- Parameter-heavy URLs that create duplicate content
- What NOT to block
- CSS and JavaScript files (Google needs these to render your pages)
- Image files you want to appear in image search
- Pages you want to rank in search results
- Pages with index,follow tags (Google needs to crawl them to see the index,follow)
AI crawler rules (2026 update)
In 2026, your robots.txt should also address AI crawlers. The recommended approach: allow search-focused AI bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and decide separately about training bots (GPTBot, ClaudeBot, Google-Extended). Use our free Robots.txt Generator to create a customized file.
Common mistakes
- Blocking your entire site with "Disallow: /" under User-agent: *
- Blocking CSS/JS resources that Google needs to render your pages
- Using robots.txt to hide pages from search (use index,follow instead)
Forgetting to include your sitemap URL at the bottom
Test your robots.txt using Google Search Console's robots.txt Tester before deploying changes. A single mistake can remove your entire site from search results.
References & Further Reading
Google Search Central
Need help putting this into practice?
Free 30-minute call to look at your specific situation.