SchemaValid
ENDE

Robots.txt Tester

Test and validate your robots.txt in seconds. This free robots.txt tester fetches a live file or checks pasted content, shows whether a URL is allowed or blocked for any crawler, and flags common mistakes — matching runs in your browser.

The paste mode runs entirely in your browser. In URL mode our server fetches only the /robots.txt file at the domain root — nothing is stored or tied to an account.

How the URL test works

The test follows the Robots Exclusion Protocol (RFC 9309), the same standard Google's crawlers use. Only one group applies to a crawler: the one with the most specific matching User-agent, falling back to the wildcard * group. A crawler with its own group ignores the * rules entirely.

Within that group, the longest matching path pattern wins — not the first one listed. That is why a broad Disallow: / can be overridden by a more specific Allow: /blog/. The tool supports the two wildcards of the standard: * for any sequence and $ to anchor the end of a path.

Blocked is not the same as removed from Google

If a URL is blocked here, crawlers will not fetch it — but it can still appear in search results if other pages link to it, usually without a description. robots.txt controls crawling, not indexing. To remove a page from the index, allow crawling and add a noindex meta tag instead.

For the full specification, see Google's robots.txt documentation: Google Search Central – robots.txt intro.

Frequently asked questions

How do I test my robots.txt?
Enter your domain to fetch the live file, or paste your robots.txt directly. The tool shows the parsed rules, flags syntax problems, and lets you enter any URL plus a crawler to see whether that URL is allowed or blocked — and exactly which rule decides.
Why is an important page being blocked?
Usually a broad Disallow pattern catches more than intended. Use the URL test with the affected path to see which rule matches. Often the fix is a more specific Allow rule, which wins because the longest matching pattern takes precedence.
What is the difference between blocked and not indexed?
Blocked (in robots.txt) means crawlers should not fetch the URL. Not indexed means it will not appear in search results. A blocked page can still be indexed if it is linked from elsewhere. For reliable removal, allow crawling and use a noindex meta tag.
What does "longest match wins" mean?
When several Allow and Disallow rules match a URL, the rule with the longest path pattern decides — not the order they appear in. If two matching rules are the same length, Allow wins. This is the core rule of the Robots Exclusion Protocol, and it is what this tester implements.
How do I check if GPTBot or ClaudeBot has access?
The AI crawler panel shows, for each known AI bot, whether the site root is allowed or blocked by the robots.txt you tested. You can also pick a specific bot in the URL test to check any individual path.
Why does Google ignore Crawl-delay?
Google never supported the Crawl-delay directive. Bing and Yandex honor it, but for Google you control crawl rate through Search Console. The tester flags a Crawl-delay that targets Google or all crawlers so you are not relying on something Google skips.
Where must the robots.txt file be located?
At the root of the host: example.com/robots.txt. It only applies to that exact host and protocol, so a subdomain like shop.example.com needs its own file. A robots.txt in a subfolder has no effect. This tool always fetches the root file.
Why do I see a different robots.txt than I expected?
Common reasons are caching (CDNs and browsers cache robots.txt), a redirect to another host, or a staging file that was deployed by mistake. The tester shows the final URL it fetched after following redirects, which helps you spot when the file served is not the one you edited.

Related tools

Robots.txt Generator

Build and validate a robots.txt with live preview — crawler rules, AI-bot control, sitemap. Free.

Sitemap Validator — Check Your XML Sitemap

Validate an XML sitemap, find a site's sitemap, test URL status, hreflang and media sitemaps. Free, no account.

AI Search Readiness Checker

Score how ready your site is for ChatGPT, Perplexity and AI Overviews.

Hreflang Generator

Build valid, reciprocal hreflang tags in HTML, XML sitemap or HTTP header format — free, no account.

Open Graph / Meta Tag Generator

Generate Open Graph and Twitter/X Card meta tags for correct link previews.

SERP Snippet Preview

Preview your Google snippet and check title & description pixel width.