SchemaValid
ENDE

Robots.txt Generator

Create a valid robots.txt with a visual editor and live preview. This free robots.txt generator handles crawler rules, AI-bot access and your sitemap — and everything runs in your browser.

Start from a preset

Crawler rules

Seconds between requests. Google ignores Crawl-delay; Bing and Yandex respect it.

AI crawlers

Choose one mode, then pick the bots. Block to keep your content out of AI training and answers (IP protection), or allow to be quotable in AI answers (AEO visibility).

Custom / advanced directives

Anything you paste here is added verbatim, including comment lines starting with #. Use it for directives the form does not cover.

Live preview

User-agent: *
Disallow:
Tip: add your sitemap URL so crawlers discover all your pages faster.
Everything runs locally in your browser. The rules and URLs you enter are never uploaded, stored, or tied to an account.

Already have a robots.txt live? Test it with the Robots.txt Tester →

robots.txt controls crawling, not indexing

A common misconception is that Disallow removes a page from Google. It does not. robots.txt only asks crawlers not to fetch a URL — but a blocked URL can still be indexed if other pages link to it, usually shown without a description because the crawler never read it.

To keep a page out of the index, let it be crawled and add a noindex meta tag (or an X-Robots-Tag header) instead. Blocking it in robots.txt actually prevents Google from seeing that noindex, so the two techniques should not be combined on the same page.

Where the file goes and how paths work

The file must be named robots.txt and live at the root of your domain — example.com/robots.txt, not in a subfolder. Each set of rules starts with a User-agent line, followed by Disallow and Allow paths. Paths are case-sensitive: /Folder/ and /folder/ are different.

Do not block your CSS or JavaScript: Google renders pages like a browser and needs those files to judge layout and content. Keep robots.txt for genuinely private or low-value areas, and remember it is a public file anyone can read at your-domain.com/robots.txt.

For the full specification, see Google's introduction to robots.txt: Google Search Central – robots.txt intro.

Frequently asked questions

What is a robots.txt file?
It is a plain text file at the root of your site that tells web crawlers which areas they may or may not fetch. It uses simple User-agent, Disallow and Allow lines. It is a request that well-behaved crawlers follow — it is not a security measure and cannot force anyone to obey it.
Does robots.txt stop a page from being indexed?
No. robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it — usually without a description. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
What is the difference between Disallow and noindex?
Disallow (in robots.txt) asks crawlers not to fetch a URL. noindex (a meta tag or header on the page) tells them not to keep it in the index. If you Disallow a page, the crawler never sees its noindex — so for reliable removal from search, use noindex and leave the page crawlable.
Should I block or allow AI crawlers?
It depends on your goal. Blocking them (GPTBot, ClaudeBot, CCBot and others) keeps your content out of AI training and answers — useful if the content is your product. Allowing them lets your brand be quoted and cited in AI answers, which is the emerging AEO/GEO opportunity. This tool makes both a one-click choice so you can decide per site.
What is Crawl-delay and does Google respect it?
Crawl-delay asks a crawler to wait a number of seconds between requests. Google does not support it at all — set the crawl rate in Search Console instead. Bing and Yandex do honor it. It is only worth adding for the crawlers that actually read it.
Where does the robots.txt file have to be?
At the very root of your domain, reachable at example.com/robots.txt. It only applies to the host and protocol it sits on, so a separate subdomain like shop.example.com needs its own robots.txt. It cannot live in a subfolder.
Should I block CSS or JavaScript?
No. Google renders your pages like a browser and needs the CSS and JavaScript to understand layout, content and mobile-friendliness. Blocking those files can hurt how your pages are evaluated. Keep robots.txt focused on genuinely private or duplicate areas.
What is the difference between robots.txt and an XML sitemap?
They are complementary. robots.txt tells crawlers where not to go; an XML sitemap lists the URLs you do want crawled and indexed. Adding a Sitemap line to robots.txt simply points crawlers to that list — this tool can add it for you.

Related tools

Robots.txt Tester

Check and validate a robots.txt: test URL access per crawler, AI-bot status, syntax warnings. Free.

AI Search Readiness Checker

Score how ready your site is for ChatGPT, Perplexity and AI Overviews.

llms.txt Generator

Generate a ready-to-use llms.txt file for your website from any URL.

SERP Snippet Preview

Preview your Google snippet and check title & description pixel width.

Schema Markup Validator

Validate a live URL or JSON-LD code: SEO score, errors, warnings and rich-results readiness.

IBAN Validator

Check whether an IBAN is valid, instantly and for free.