robots.txt generator robots.txt
Which bots may do what? The generator builds your robots.txt in four steps: generate, check, test, compare. Everything local, only your site's robots.txt is read.
Experimental. The tools are based on public statements by AI providers, current studies, published assessments by AI and SEO experts and hands-on experience of site owners. AI platforms do not disclose their criteria, so the results are approximations, experience and observations.
1What is it?
Generate: pick a template (WordPress, WooCommerce, Shopware, Joomla, staging), select bots per group, sitemap line and comments in your language. Check: the current robots.txt for typos, blocked CSS, missing sitemap and contradictions. Test: check a URL against the rules, by the Google rule (longest matching path wins). Compare: old against new as a diff.
2Why it matters
A typo like "Disalow" is silently ignored. A Disallow: / left over from staging removes the whole site from search. And whoever blocks all AI bots also disappears from AI search, not just from training.
3What TagSpion checks
| Check | Rating |
|---|---|
| Bot groups | search engines, AI search and fetches, data collectors (training), SEO tool crawlers, archive. Named by function, no judgement. |
| Check | typos, unknown directives, Crawl-delay, noindex in robots.txt, CSS or JS blocked, wp-admin without admin-ajax, Disallow: / for all, no sitemap, over 500 KB |
| Test | allowed or blocked per bot, with the line that applies |
4How to fix it
- Save the file as robots.txt and put it in the root of the domain: https://www.example.com/robots.txt
- After uploading, check the most important URLs in the "Test" tab.
- Remove the staging template before going live.
# robots.txt - AI search allowed, training blocked User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # Training bots (your decision, no effect on visibility) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / # AI search bots stay allowed: OAI-SearchBot, PerplexityBot, Claude-SearchBot Sitemap: https://www.example.com/sitemap.xml
5Frequently asked questions
Should I block AI bots?
That is your decision. Blocking training (GPTBot, ClaudeBot, Google-Extended) costs no visibility. Blocking AI search (OAI-SearchBot, PerplexityBot, Claude-SearchBot) takes you out of AI answers.
Do all bots respect robots.txt?
The big providers do, by their own account. robots.txt is a request, not a lock. If you want to block for sure, you need server rules.
What are the signals (Content-Signal, Content-Usage)?
Proposals by Cloudflare and the IETF to declare usage. Not a standard, in testing. TagSpion shows them, they have no effect on the grade.