← All tools

Robots.txt generator

Build robots.txt rules and test any path against them.
Start from
Rules (user-agent · allow or disallow · path)
Sitemap URL
Crawl-delay (seconds, optional)
Google ignores this. Bing and some others respect it.
Test a URL path
✕ Blocked by “Disallow: /admin/”
All checks pass
  • Note: robots.txt controls crawling, not indexing. Use noindex to keep a page out of search results.
  • Pass: 2 user-agent groups, 4 rules.
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /search/help
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
● All goodRobots.txt generatorLocal-only

What it’s good for

Control crawlers without blocking your own site.

  • Block private areasKeep admin, search and API paths out of crawls.
  • Manage AI crawlersOpt out of training crawlers by user-agent.
  • Test before deployingSee whether a path is allowed for each bot.

Questions

The short answers.

Does Disallow remove a page from search?

No. It stops crawling. Use a noindex tag to keep a page out of results.

How are conflicts resolved?

The longest matching path wins. On a tie, Allow wins.

Where does the file go?

At the root: https://yourdomain.com/robots.txt.

Which group does a bot follow?

Only the group that names it. A bot with its own User-agent group ignores the * group entirely, so repeat any shared rules in that group. The tester follows the same logic.

Do all crawlers obey robots.txt?

Reputable search engines and the major AI crawlers say they do, but it's a request, not access control. Protect private content with authentication, not robots.txt.

See all →