AI Crawler Check
Enter a domain and a path. The tool reads the site's robots.txt and shows, bot by bot, whether each AI crawler is allowed to fetch that path, and which rule decides it.
AI visibility
An AI engine can only cite a page its crawler was allowed to read.
Each AI company runs several bots, each with its own job, and robots.txt treats them one at a time. A rule written for one bot can block another, and a general
Allow can be overridden by a group aimed at a single crawler. This check applies the rules the way the robots.txt standard (RFC 9309) describes: the most specific group for each bot, then the longest matching rule, with Allow winning a tie.Check a site
Or paste a robots.txt file
Nothing checked yet.
Bot by bot
Results
Rules read from robots.txt. A firewall or CDN can still turn a bot away before it reads them.
| Bot | Run by | Its job | This path |
|---|
Bot names and jobs were checked against each company's crawler documentation on 2 October 2026. They change, so read the company's own page before you edit a rule. Whether to allow training bots is a business decision.
Rules are half of it
This check reads what robots.txt allows. To see which bots visit, read your server logs. To change the rules, generate them bot by bot.
Frequently asked questions
What does the AI crawler check do?
It reads a site's robots.txt and works out, for each AI crawler, whether the rules allow it to fetch the path you enter, and which rule decides it. It doesn't fetch the page as each bot and can't see blocks set in a firewall or CDN.
Does "Allowed" mean AI engines can read my page?
No. It means robots.txt doesn't stop that bot. A firewall, CDN or bot-protection setting can still turn it away, and many AI crawlers have not been observed to run JavaScript, so content that loads by script may be missing from what they read.
Should I block AI training bots?
That is a business decision. Blocking a training bot such as GPTBot keeps your content out of training, and OpenAI says ChatGPT search is not affected, because that uses OAI-SearchBot. Read each company's documentation before you change a rule.
Why do some bots carry a note about user requests?
ChatGPT-User, Claude-User and Perplexity-User open a page when a person's request needs it. Anthropic says Claude-User follows robots.txt. The other two act for a person, and robots.txt may not apply to them.
Does the check look at llms.txt?
No. llms.txt is a proposal from 2024, not a standard, and Google says such files aren't needed for Search or its AI features. Treat it as optional: robots.txt is what crawlers read for access.
Is there a free way to see which AI bots visit?
Yes. The free AI crawler log analyser reads your server logs and shows which AI bots came and what they fetched. In SEORCE, Web Analytics and AI Beacon show visits from AI crawlers and other bots over time.
Free · no account
SEORCE for Chrome, Edge and Firefox
Audits the page you are on: headings, meta tags, schema, indexability, and whether AI crawlers can read it. Export the result as a PDF. Free, with no account.
It audits one page at a time and does not track AI answers. About the extension
Where you stand in AI answers
This tool checks the inputs. AI Beacon shows the answers.
SEORCE AI Beacon tracks whether your brand is named and cited across 7 AI engines + Google AI Overview, AI Mode & Search. It shows the exact page behind each citation, and whether it was cited or only consulted.
3,000+ users · Free forever plan, no card needed