Robots.txt Generator
SEODefine crawling policies and control search engine access to your website
Presets
Blocked by “Disallow: /admin/”
robots.txt controls crawling, not indexing — use a noindex meta tag to keep a page out of results.
Rule group
* for all, or e.g. Googlebot
Ignored by Google
robots.txt
User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /cart Disallow: /checkout Disallow: /search Disallow: /*?*sort= Sitemap: https://example.com/sitemap.xml