The Rankyfy AI Crawler Access Checker tests whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot can read your website. It reads your robots.txt file and then requests your page as each bot, so it also catches firewall blocks. It covers 16 crawlers and is free.
AI search bots can read your site. 2 training bots are blocked.
| Bot | Company | Purpose | robots.txt | Live fetch | Verdict |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search | Allowed | OK | Can read |
| Claude-SearchBot | Anthropic | AI search | Allowed | OK | Can read |
| PerplexityBot | Perplexity | AI search | Not mentioned | OK | Can read |
| ChatGPT-User | OpenAI | User request | Allowed | OK | Can read |
| GPTBot | OpenAI | Training | Blocked | OK | Cannot read |
| CCBot | Common Crawl | Training | Blocked | Blocked | Cannot read |
Showing 6 of 16 crawlers, grouped by purpose: AI search, user requests, then training.
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: /
Being readable is the first step. Next, see whether ChatGPT and Google AI recommend you.
Fetching your home page as GPTBot…
You have used today's 10 free checks. They reset at midnight.
We could not reach that website. Check the address and try again.
An AI crawler is a program that fetches web pages for an AI company. Some collect text to train models. Some build the search index that AI answers cite. Some fetch a single page at the moment a person asks a question.
The robots.txt column shows what your file tells each bot. The live fetch column shows what your server did when asked. A bot must pass both to read your site. “Not mentioned” means the bot follows your general rule.
A firewall or content network can refuse AI bots even when robots.txt allows them. A page that needs JavaScript to show its text can also look empty, because AI crawlers usually read only the raw HTML.
Blocking training bots keeps your text out of model training. Blocking search and user-triggered bots removes you from AI answers. A business that wants customers should usually allow the search and user-triggered bots.
Yes. You get 10 checks a day with no account.
No. Access only lets AI read your site. Whether AI names your business depends on what it finds there and elsewhere. Rankyfy AI Visibility measures that.
The major AI companies say their crawlers follow it. Robots.txt is a request, not a lock. A firewall rule is the lock.
It is optional. No major AI company has confirmed that it reads the file. The Rankyfy llms.txt Generator explains what it is good for.
The test uses each bot's name from Rankyfy's servers. A firewall that checks the bot's IP address may answer the real bot differently.