Visually build robots.txt rules and download. SEO basics.
Rule groups
📄 robots.txt

How to use

  1. Pick a preset (allow all / block all / standard / block AI).
  2. Add/edit groups (User-agent + Allow/Disallow rules).
  3. Enter sitemap URLs.
  4. Download the file and upload to site root (/robots.txt).
  5. Test in Google Search Console.

Key features

  • 4 presets (allow / block / standard / block AI bots)
  • User-agent autocomplete (Googlebot, GPTBot, CCBot, etc.)
  • Multiple rule groups
  • Sitemap + Crawl-delay support
  • Live preview + download/copy

Use cases

  • SEO base setup — first step for new sites
  • Block admin pages — /admin, /api indexing prevention
  • Staging sites — block dev environment
  • AI opt-out — block GPTBot, CCBot, Claude-Web
  • Image indexing — allow/block Googlebot-Image

About robots.txt

robots.txt is a text file at site root (e.g., https://pxltool.com/robots.txt) telling crawlers which pages to crawl. Not legally binding but major engines respect it. User-agent: * = all bots, Disallow: / = block everything, Allow: = allow specific paths. Not absolute blocking — for real security use authentication or IP blocks. Since 2023, blocking AI training bots (GPTBot, ClaudeBot) is a trend.

FAQ

Disallowed URL shows in search results.

Disallow blocks crawling but not indexing. URLs found via other site links may still be indexed. Use noindex meta tag for full block.

Google ignores my robots.txt.

Cached up to 24h. Use the Google Search Console robots.txt tester to force re-crawl.

Korean engines (Naver, Daum)?

Use Naverbot, Daumoa User-agents — included in autocomplete.

Free?

Yes.