Robots.txt Generator & Tester
Build a correct robots.txt in seconds — choose what crawlers may access, list your sitemaps, block AI training bots, then test any URL to see if it’s allowed. Free and private.
Free · No sign-upHow to use Robots.txt Generator & Tester
- 1. Start from a preset (allow all, WordPress, online store, block AI bots) or set the rule for all crawlers.
- 2. List the paths to disallow and any exceptions, your sitemap URLs and optional crawl-delay.
- 3. Tick the AI crawlers to block and add rules for specific bots if you need them.
- 4. Test a few URLs in the tester, then copy or download robots.txt and upload it to your site’s root.
Frequently asked questions
Where do I put robots.txt?
At the root of your domain, so it loads at https://example.com/robots.txt. Each subdomain (such as shop.example.com) needs its own file. Crawlers ignore robots.txt files in subfolders.
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing — a blocked URL can still appear in results if other sites link to it. To remove a page, let Google crawl it and add a noindex meta tag, or password-protect it.
How do I block AI crawlers like GPTBot?
Tick the AI crawlers you want to block. Each gets its own User-agent group with Disallow: /. For example GPTBot and ClaudeBot are used for model training, while Google-Extended controls Gemini training without affecting Google Search.
Which rule wins when Allow and Disallow both match?
Under RFC 9309, which Google and Bing follow, the most specific (longest) matching rule wins, and Allow wins a tie. A crawler only obeys the group that names it most specifically, falling back to User-agent: *.
What do * and $ mean in robots.txt?
* matches any sequence of characters and $ marks the end of the URL. For example Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*?sort= blocks sorted listing pages.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay and adjusts its crawl rate automatically. Bing and Yandex honor it, so it can still help if those crawlers put load on your server.