robots.txt guide

What is robots.txt?

robots.txt is a text file at the root of your website that tells web crawlers (like Googlebot) which pages or files they can and cannot request from your site. It's a standard used by websites to communicate with web robots and search engine crawlers.

Why Use robots.txt?

• Prevent indexing of private areas: Block admin panels, staging sites, and internal tools
• Save crawl budget: Help search engines focus on important content
• Reduce server load: Prevent crawlers from accessing resource-heavy pages
• Block duplicate content: Prevent indexing of print versions, filtered results, etc.

Basic Syntax

User-agent Directive

Specifies which crawler the rules apply to. Use * for all crawlers.

User-agent: *        # Applies to all crawlers
User-agent: Googlebot  # Only applies to Googlebot
User-agent: Bingbot    # Only applies to Bingbot

Disallow Directive

Blocks crawlers from accessing specific paths.

Disallow: /admin/        # Blocks entire /admin/ directory
Disallow: /private.html   # Blocks specific file
Disallow: /               # Blocks entire site (use carefully!)

Allow Directive

Overrides Disallow for specific paths (supported by major search engines).

Disallow: /admin/
Allow: /admin/public.html  # Allows this specific file

Sitemap Directive

Tells crawlers where to find your XML sitemap.

Sitemap: https://example.com/sitemap.xml

Complete Example

# robots.txt for example.com

# Allow all crawlers full access by default
User-agent: *
Allow: /

# Block specific directories
Disallow: /admin/
Disallow: /api/
Disallow: /temp/
Disallow: /search?

# Block specific file types
Disallow: /*.pdf$
Disallow: /*.zip$

# Allow specific content within blocked areas
Allow: /admin/help.html

# Declare sitemap location
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xml

Common Patterns

Block All Crawlers

User-agent: *
Disallow: /

Allow All Crawlers

User-agent: *
Disallow:

Block Specific Crawler

User-agent: BadBot
Disallow: /

User-agent: *
Allow: /

Block URL Parameters

# Block all URLs with query strings
Disallow: /*?

Best Practices

• Place at root: Must be at /robots.txt
• Use correct syntax: One directive per line, case-sensitive
• Test before deploying: Use the Robots.txt Tester
• Don't rely on it for security: It's a guideline, not a barrier
• Keep it simple: Complex rules are harder to maintain

Important Limitations

• Not a security mechanism: Malicious bots can ignore robots.txt
• Doesn't remove indexed content: Use noindex meta tags or Google Search Console for that
• Publicly visible: Anyone can see what you're trying to hide
• Not all crawlers support all directives: Allow directive isn't universal

Related Tools

• Robots.txt Tester - Test your rules

FAQ

Q: Does robots.txt prevent indexing?
A: No, it prevents crawling. Pages can still be indexed if linked from other sites. Use noindex meta tags to prevent indexing.

Q: Can I have multiple robots.txt files?
A: No, only one at the domain root. For subdomains, each needs its own robots.txt.

Q: How long until changes take effect?
A: Crawlers check robots.txt periodically. Major search engines typically respect changes within days.

Q: What's the difference between Disallow and noindex?
A: Disallow (in robots.txt) tells crawlers not to fetch the page. Noindex (meta tag) tells search engines not to include the page in search results.