Robots.txt Generator

The Robots.txt Generator creates a valid robots.txt file for your website with allow/disallow rules, user-agent targeting, sitemap references, and crawl-delay directives. Robots.txt is the file search engines check first when crawling your site to determine which pages they can or cannot access. Used correctly, it prevents crawlers from wasting resources on admin pages, filter URLs, and other content that shouldn\u2019t appear in search. Used incorrectly, it can inadvertently hide your entire site from search engines.

Configure rules via structured forms: specify user agents (* for all, or specific like Googlebot, Bingbot), add disallow patterns for paths to block, allow patterns for exceptions, set crawl-delay (mostly for Bing), and reference your sitemap URL. The tool outputs standards-compliant robots.txt ready to upload to your site root. Common presets included: standard WordPress robots.txt, e-commerce with blocked checkout and cart, single-page app avoiding crawl of query variations. All generation runs in your browser.

Robots.txt follows the Robots Exclusion Protocol (originally 1994). Google formalized it in 2019 as a proposed RFC. Placed at the root: https://example.com/robots.txt.

Basic syntax:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Sitemap: https://example.com/sitemap.xml

Directives:
- User-agent: identifies which crawler the following rules apply to. * means all. Googlebot, Bingbot, Slurp, DuckDuckBot are specific.
- Disallow: path prefix or pattern to block
- Allow: exception to Disallow (specific path inside a disallowed directory)
- Crawl-delay: seconds between crawler requests (ignored by Google; respected by Bing, Yandex)
- Sitemap: URL of sitemap file (reference only, not rule)

Path matching:
- Disallow: /admin/ blocks /admin/ and everything inside
- Disallow: /page blocks /page, /pages, /page.html (prefix match)
- Disallow: /*.pdf$ blocks URLs ending with .pdf (wildcard)
- Disallow: / blocks everything (emergency use only)
- Disallow: (empty) allows everything for this user-agent

Google supports * and $ wildcards. Older crawlers may not.

Rule precedence:
- Most specific rule wins
- Allow beats Disallow when paths are equally specific
- Different User-agent blocks are independent; a specific user-agent matches first if named

Common patterns:
- Allow all: User-agent: * / Disallow: (empty)
- Block all: User-agent: * / Disallow: /
- Block admin: User-agent: * / Disallow: /admin/
- Block search parameters: Disallow: /*?s= (block internal search URLs)
- Block file types: Disallow: /*.pdf$ (block PDF indexing)

Important caveats:
- Robots.txt is a request, not an enforcement. Well-behaved crawlers respect it; malicious ones ignore it
- Blocking URLs in robots.txt does NOT deindex them. Google may keep previously indexed URLs. For deindexing, use noindex meta tag or X-Robots-Tag header
- Robots.txt blocks affect crawling but may not affect indexing if the page is linked from elsewhere
- Don\u2019t use robots.txt to hide sensitive content — it reveals the existence of paths. Use authentication or IP restriction
- Syntax errors can silently break robots.txt — always validate

Size limits: Google parses first 500 KiB of robots.txt. Anything beyond is ignored.

Case sensitivity: paths are case-sensitive on most servers. /Admin/ is different from /admin/ on Unix-based hosts.

Compared to writing robots.txt by hand, this tool provides structured forms and validates against common mistakes. Hand-writing is possible but error-prone.

Compared to CMS auto-generated robots.txt (WordPress, Shopify), this tool gives manual control for sites without auto-generation or when CMS defaults don\u2019t fit.

Compared to Google\u2019s robots.txt tester, this tool is for generation; the tester is for validation. Generate here, test there.