Skip to main content
tlify
PDFImageUtilityMediaTools
tlify© 2026 · Browser-side processing · No account needed
PrivacyTermsCookiesContact
HomePDFImageMediaMore
R
03·Utility/robots-txt-generator

Robots.txt Generator

in-browser
⌘K
How it works
  1. Add user-agent rules with disallow and allow paths.
  2. Set the sitemap URL and optional crawl delay.
  3. Copy or download the generated robots.txt file.

Related

‹/›Meta Tag GeneratorHHTML to PDF{}JSON Formatter
Disallow
Allow

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
A file that tells search engine crawlers which pages they can or cannot access on your site.
In the root of your website, accessible at yourdomain.com/robots.txt.
It blocks crawling, not indexing. To prevent indexing, use a noindex meta tag instead.