Bot directory Bot Directory
Who is knocking? The directory lists all known bots with their function, whether they are allowed on your page and where the official documentation is.
Experimental. The tools are based on public statements by AI providers, current studies, published assessments by AI and SEO experts and hands-on experience of site owners. AI platforms do not disclose their criteria, so the results are approximations, experience and observations.
1What is it?
Every row shows the user agent, the operator, the group (search engine, AI search and fetches, data collector for training, SEO tool crawler, archive), the status according to your robots.txt and the link to the provider's documentation.
2Why it matters
Many block "the AI" wholesale and do not notice they are also locking out AI search. Whoever knows the bots can decide precisely.
3What TagSpion checks
| Check | Rating |
|---|---|
| Status per bot | allowed or blocked for the current page, by RFC 9309 |
| Groups | named by function, no judgement |
All bots at a glance
Search engines
Classic search, basis for AI answers
| Bot (user agent) | Operator | Docs |
|---|---|---|
Googlebot | Google · AI Overviews | Docs ↗ |
Bingbot | Bing · Copilot | Docs ↗ |
Applebot | Apple · Siri | Docs ↗ |
AI search and fetches
Reads pages for AI answers and on user request
| Bot (user agent) | Operator | Docs |
|---|---|---|
OAI-SearchBot | OpenAI · ChatGPT Search | Docs ↗ |
ChatGPT-User | OpenAI · ChatGPT | Docs ↗ |
Claude-SearchBot | Anthropic · Claude Search | Docs ↗ |
Claude-User | Anthropic · Claude | Docs ↗ |
PerplexityBot | Perplexity · Search | Docs ↗ |
Perplexity-User | Perplexity | Docs ↗ |
DuckAssistBot | DuckDuckGo | Docs ↗ |
MistralAI-User | Mistral · Le Chat | Docs ↗ |
Data collectors (training)
Collects content for model training
| Bot (user agent) | Operator | Docs |
|---|---|---|
GPTBot | OpenAI | Docs ↗ |
ClaudeBot | Anthropic | Docs ↗ |
Google-Extended | Google · Gemini | Docs ↗ |
Applebot-Extended | Apple · Apple Intelligence | Docs ↗ |
meta-externalagent | Meta · Meta AI | Docs ↗ |
CCBot | Common Crawl | Docs ↗ |
Bytespider | ByteDance | – |
SEO tool crawlers
Collects backlinks and data for SEO tools
| Bot (user agent) | Operator | Docs |
|---|---|---|
AhrefsBot | Ahrefs | Docs ↗ |
SemrushBot | Semrush | Docs ↗ |
MJ12bot | Majestic | Docs ↗ |
DotBot | Moz | Docs ↗ |
Archive
Archives pages (Wayback Machine)
| Bot (user agent) | Operator | Docs |
|---|---|---|
ia_archiver | Internet Archive | Docs ↗ |
4How to fix it
- Decide per group and implement it in the robots.txt generator.
- Whether a bot is genuine is only shown by comparing with the providers' official IP lists, not by the user agent.
5Frequently asked questions
Why is no group called "bad bots"?
Because TagSpion does not judge companies. An SEO crawler is useful for one person and a burden for another. You decide.
A bot is missing?
Write to tagspion@outlook.com with a link to the provider's documentation.