robots.txt Generator
Build a valid robots.txt file with user-agent groups, Allow and Disallow rules, crawl-delay and a sitemap. Everything runs in your browser — your data is never sent anywhere and nothing is stored.
Note: robots.txt is a crawl directive, not a security control. Compliant crawlers obey it, but it does not block access and bad bots can ignore it. Use authentication or noindex to protect private content.
How to use the robots.txt generator
- Pick a quick preset, or choose Custom to start from scratch.
- Set the default crawl policy and edit the user-agent groups — add Disallow and Allow paths, and extra groups for specific crawlers.
- Optionally add a crawl-delay and your sitemap URL.
- Copy the result or download it, then upload it to the root of your site as
/robots.txt.
Paths are matched from the start of the URL path and should begin with /. An empty Disallow: allows everything, while Disallow: / blocks the whole site.
More PAYATE tools
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of your site (/robots.txt) that tells search-engine crawlers which parts of your site they may or may not request. It uses simple User-agent, Allow and Disallow directives.
Where do I put the robots.txt file?
It must live at the root of your domain — for example https://www.example.com/robots.txt. A robots.txt in a subfolder is ignored. Each subdomain and protocol needs its own file.
Does robots.txt keep pages private or secure?
No. robots.txt is a crawl directive, not a security control. Well-behaved crawlers obey it, but it does not block access and malicious bots can ignore it. A disallowed URL can still be indexed if linked elsewhere. Use authentication or noindex for real protection.
What is the difference between Allow and Disallow?
Disallow tells a crawler not to request matching paths; Allow creates an exception that permits a path inside a disallowed area. An empty Disallow: means nothing is blocked (allow all).
Should I add my sitemap here?
Yes, it helps. A Sitemap: line pointing to your XML sitemap's full URL lets crawlers discover your pages faster. It is independent of the user-agent groups and is usually placed at the end of the file.
What does User-agent: * mean?
The * wildcard matches all crawlers that do not have their own named group. You can add specific groups (for example Googlebot) with their own rules, and a catch-all * group for everyone else.