robots.txt Generator
Build a robots.txt file with per-user-agent allow and disallow rules, plus a sitemap reference.
robots.txt
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
What is robots.txt Generator?
robots.txt Generator constructs standard Robots Exclusion Protocol files for website crawlers. It allows you to define multiple user-agent rule blocks (targeting all crawlers with * or specific bots like Googlebot), specify multi-line Disallow and Allow path directives, and append your XML sitemap URL. The generated file is formatted with standard line breaks and ready for immediate download.
How to use robots.txt Generator
- 1Add or edit a rule block for each crawler (use * for all search engines or specific names like Googlebot).
- 2Enter paths to disallow (e.g. /admin/) or allow, one path per line.
- 3Add your XML sitemap URL and click Copy or Download robots.txt.
Key features
Multi-User-Agent Blocks
Configure independent crawl rules for global crawlers (*) or specific search engine bots.
Multi-Path Allow & Disallow
Specify multiple directories or paths per rule block with automatic whitespace trimming.
XML Sitemap Reference
Automatically appends the Sitemap directive at the bottom of the file for search crawler discovery.
One-Click Export
Copy text directly or download as a ready-to-deploy robots.txt text file.
Examples & use cases
Standard Site Configuration
Disallowing admin section while allowing public content and declaring sitemap
User-agent: *, Disallow: /admin/, Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Multi-Crawler Directives
Different directives for all bots and specific media scrapers
User-agent: *, Disallow: /private/ User-agent: Googlebot-Image, Disallow: /temp/
User-agent: * Disallow: /private/ User-agent: Googlebot-Image Disallow: /temp/
Frequently asked questions
Related tools
Generate a valid sitemap.xml file from a list of URLs.
Generate SEO, Open Graph and Twitter Card meta tags together.
Convert text into a clean, lowercase, URL-friendly slug.
Normalize a URL and strip tracking parameters (utm_*, fbclid, etc.).
Build a campaign tracking URL with UTM parameters for analytics.