OpusDesk Hub What this tool does
Draft robots.txt crawler rules. The generator writes user-agent groups and the entered Allow/Disallow paths, plus optional sitemap and crawler fields exposed by the interface. The output is plain text; it is not installed or tested on a server.
How to use Robots.txt Generator
- Prepare the input. Choose the user agent and enter exact paths.
- Run or configure the tool. Add only applicable optional sitemap/directive fields.
- Check and use the output. Review the resulting text and test the rules before publishing.
How this tool works
The generator writes user-agent groups and the entered Allow/Disallow paths, plus optional sitemap and crawler fields exposed by the interface. The output is plain text; it is not installed or tested on a server.
Worked example
Input: Disallow path /qa-private/
Output: Disallow: /qa-private/
Limits, assumptions and interpretation
robots.txt is not access control and must not protect secrets or private documents.
- Crawler support differs for optional directives; a generated rule is not evidence that a crawler obeyed it.
- A blocked page can still be referenced in search. Blocking crawling can prevent a crawler from reading a page’s noindex directive.
- Test matching rules and publish at the origin’s /robots.txt; do not accidentally block required tools.
Supported inputs and limits
- Disallow path /qa-private/
- robots.txt is not access control and must not protect secrets or private documents.
- Crawler support differs for optional directives; a generated rule is not evidence that a crawler obeyed it.
- A blocked page can still be referenced in search. Blocking crawling can prevent a crawler from reading a page’s noindex directive.
- Test matching rules and publish at the origin’s /robots.txt; do not accidentally block required tools.
Sources and specifications
Frequently asked questions
What does this tool actually do?
The generator writes user-agent groups and the entered Allow/Disallow paths, plus optional sitemap and crawler fields exposed by the interface. The output is plain text; it is not installed or tested on a server.
What should I check before using the result?
robots.txt is not access control and must not protect secrets or private documents. Crawler support differs for optional directives; a generated rule is not evidence that a crawler obeyed it. A blocked page can still be referenced in search. Blocking crawling can prevent a crawler from reading a page’s noindex directive. Test matching rules and publish at the origin’s /robots.txt; do not accidentally block required tools.
Is information sent to a server?
Tool inputs are processed in this browser. This product does not use analytics or send your input to an external API. Clicking an external website link still visits that website.