3. Can the answer engines read you
What this is
The crawlers everybody blocks are the ones that train models. They are not the ones that decide whether an assistant can quote you. Those are different, they are named differently at every vendor, and blocking them by accident is easy — Apple, for instance, follows your Googlebot rule when you have not named Apple.
What we found on vitadefence.com
Every crawler that decides whether an AI can quote you can already reach this site. Nothing to fix here.
All 6 answering crawlers can reach vitadefence.com. Your robots file answered HTTP 200.
| Crawler | Whose | Can reach vitadefence.com |
|---|---|---|
| OAI-SearchBot | OpenAI | yes |
| Claude-SearchBot | Anthropic | yes |
| Claude-User | Anthropic | yes |
| Googlebot | yes | |
| PerplexityBot | Perplexity | yes |
| Applebot | Apple | yes |
Not checked, and said so rather than assumed: Google-Extended, meta-externalagent, Bingbot, CCBot, Amazonbot, Bytespider.
Measured on https://vitadefence.com.
The file at packs/03-ai-crawlers/robots-block.txt says the same thing for vitadefence.com, so you can check this yourself rather than take our word for it.
Nothing to do here yet
There is no step to follow in this chapter for vitadefence.com today. Nothing has been invented to give you something to do.
When this starts to matter
Every crawler that decides whether an assistant can quote you can already reach this site.