Your site can rank well on Google and still be invisible inside ChatGPT, Perplexity or Claude. Often the reason is one line in a file most owners never open: robots.txt. The short answer is simple. Allow the AI search crawlers, decide separately on the training crawlers, and keep Googlebot and Bingbot open.
This robots.txt for AI crawlers guide shows which bot does what, the exact rules to copy, the mistakes that quietly lock AI tools out, and how to confirm the fix worked.
Want AI tools to find and cite your pages? Start with one rule
AI companies no longer run one crawler. They run separate bots for search, for model training and for live visits a user asks for. Blocking one does not block the others.
That split is good news. You can stay visible in AI answers while keeping your content out of training sets. The rule of thumb for most service businesses:
Allow AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.
Treat training crawlers such as GPTBot, ClaudeBot and Google-Extended as a business decision.
Never block Googlebot or Bingbot. Google AI Overviews run on Googlebot, and Microsoft Copilot leans on the Bing index.
Get those three calls right and the rest is housekeeping.
Which AI crawler does what? Search bots, training bots and user fetchers
The names look alike, so owners block the wrong one all the time. Here is what each major user agent does, based on each vendor's own crawler documentation.
User agent | Company | Job | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers | Your pages stop appearing in ChatGPT search answers |
GPTBot | OpenAI | Collects content that may train OpenAI models | Future content stays out of training; ChatGPT search is not affected |
ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT about it | OpenAI says robots.txt rules may not apply |
Claude-SearchBot | Anthropic | Crawls to improve search results in Claude | Weaker visibility in Claude answers |
ClaudeBot | Anthropic | Collects content for training Anthropic models | Opts you out of training |
Claude-User | Anthropic | Fetches a page when a user asks Claude a question | Claude cannot read the page for that user |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity; not used for training | Perplexity may show only your domain, headline and a brief summary |
Perplexity-User | Perplexity | Visits a page to answer a user's question | Perplexity says this fetcher generally ignores robots.txt |
Google-Extended | Control token for Gemini training and grounding; not a crawler | No effect on Google Search or ranking | |
Applebot-Extended | Apple | Control token for Apple AI training | Applebot keeps crawling for Siri and Spotlight |
CCBot | Common Crawl | Builds an open web archive that many AI models train on | Opts you out of future Common Crawl snapshots |
Notice the pattern. Search bots bring visibility. Training bots take content. User fetchers sit in between, and two vendors say robots.txt may not stop them.
Does blocking Google-Extended pull you out of AI Overviews?
No. This mix-up is common. Google's crawler documentation says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
AI Overviews and AI Mode are Google Search features, so Googlebot controls them. Google-Extended only governs whether Google may use your content to train Gemini models and to ground answers in Gemini Apps and Vertex AI.
So a business that wants AI Overview citations but no Gemini training can block Google-Extended safely. Blocking Googlebot to "stop AI" would wipe out your organic traffic too. If you are weighing these two surfaces, our post on ChatGPT search vs Google AI Overviews explains how each one picks sources.
How do you write robots.txt rules for AI crawlers?
Each crawler obeys the group that names it. If no group names it, it falls back to the wildcard group for all user agents. Order does not matter, but clear grouping keeps the file readable for the next person who edits it.
Here is a starting point for a service business that wants AI search visibility and no model training:
# AI search: allow so AI tools can find and cite pages
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# AI training: opted out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlSwap the admin path and sitemap URL for your own. Then roll it out in a set order:
Download your current robots.txt and save a copy before you touch it.
List every private area that must stay closed, such as logins, carts and staging folders.
Add the AI search groups first, then decide on the training groups with whoever owns your content.
Publish the file at the root of each host. A blog on a subdomain needs its own file.
Recheck after a day. OpenAI says ChatGPT search takes about 24 hours to reflect a robots.txt change.
Want to keep only part of the site out of training? Scope the training group to paths instead of the root. A research library can stay closed while your service pages remain open to every bot.
What mistakes quietly block AI search bots?
Most lockouts are accidents. Nobody decided to vanish from ChatGPT. A setting did it for them.
A blanket wildcard block. A group for all user agents with Disallow: / also blocks OAI-SearchBot unless that bot has its own group.
Firewall or CDN bot settings. Some CDNs and security tools offer one-click AI bot blocking. It can return errors to AI crawlers even when robots.txt says allow.
Staging rules that went live. A redesign launches with the dev site's "block everything" file. Our website migration SEO checklist covers this check.
A robots.txt that errors. If the file returns a server error, Google may pause crawling the site. AI crawlers have even less patience.
Content only JavaScript can show. Many AI crawlers read raw HTML. Prices, services and FAQs that load only through scripts may never be seen.
Copy-pasted block lists. Lists shared online often lump search bots in with scrapers. Paste one in and your AI search visibility goes with it.
One more thing to remember. Robots.txt is a request, not a lock. Anything can pretend to be GPTBot. Real protection for private content needs logins or server rules.
How can you check that AI crawlers can reach your pages?
Writing the rules is half the job. Proving they work is the other half, and it does not take long.
Open the live file. Visit yourdomain.com/robots.txt in a browser and read each group out loud. Confirm it loads without an error.
Test a page against each bot. Use a robots.txt testing tool and run your homepage and top service page against OAI-SearchBot, Claude-SearchBot and PerplexityBot.
Read your server logs. Search for each user agent name. Hits with a 200 status mean the bot got in. A run of 403 errors points at a firewall.
Verify the visitor is real. OpenAI, Perplexity and Google publish IP ranges for their bots. A "GPTBot" from an unlisted IP is an impostor.
Ask the AI tools directly. A week or two later, run the prompts your buyers use. Our guide on how to check if ChatGPT mentions your business walks through that test.
Log the date of every change. When citations move, you will know which edit moved them.
Should a service business block AI training crawlers at all?
There is no single right answer. It comes down to what your content is worth if someone else reuses it.
Your situation | AI search bots | AI training bots | Why |
|---|---|---|---|
Local or home service business with marketing pages | Allow | Usually allow | Models that know your brand can describe you accurately from memory |
B2B firm with original research or paid guides | Allow | Block the paid or research paths | Keeps the asset you sell out of training while service pages stay open |
Publisher or membership site | Allow selected sections | Block | Your content is the product |
Staging, test or private host | Block | Block | Nothing on it should be quoted anywhere |
Keep an eye on the standards, too. Google's crawler documentation says it plans to support a shared machine-readable AI training preference signal in early 2027. Your file may need a small update when that lands.
Is crawler access enough to get cited in AI answers?
No. Access is the entry ticket, not the win. An AI tool that can read your site still needs a reason to quote it.
That reason comes from clear service pages, plain facts stated in full sentences, matching business details across the web, and structured data that names who you are. Our post on GEO vs AEO vs SEO breaks down how those layers fit together.
Structure shows up in real results.
SteamPro Cleaning & Restoration offered eleven services while one homepage tried to cover them all. After the site structure and booking path were rebuilt, the SteamPro case study reports +268% organic traffic and +154% booked jobs. Pages that are easy to crawl and easy to understand serve Google and AI tools alike.
Setup NYC had a similar problem across two audiences. Separate residential and enterprise paths produced +286% organic traffic and +163% enterprise inquiries, as the Setup NYC case study shows. These are general SEO outcomes, not robots.txt promises.
When should you hand crawler access to a technical SEO team?
A five-line robots.txt is a job you can do yourself. Bring in help when the setup has more moving parts than one file.
Your site runs on several subdomains or behind a CDN with its own bot rules.
Server logs show AI bots hitting errors and nobody knows why.
A migration or redesign is coming and the robots.txt has to survive it.
You show up on Google but not in ChatGPT, Perplexity or Claude answers.
Our technical SEO services cover crawl access, server responses, rendering and log checks in one pass. If the bigger goal is being named in AI answers, our AI SEO services and ChatGPT optimization work build on that access. Not sure where the problem sits? An SEO audit finds it first.
FAQ
Should I block GPTBot in robots.txt?
Block GPTBot only if you do not want OpenAI to train models on your content. OpenAI says blocking GPTBot does not affect ChatGPT search. To stay visible in ChatGPT answers, keep OAI-SearchBot allowed in its own robots.txt group.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI models. OAI-SearchBot finds pages to show and link in ChatGPT search answers. Each one follows its own robots.txt group, so you can allow one and block the other.
Does blocking Google-Extended affect AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are a Search feature controlled by Googlebot. Google-Extended only covers Gemini training and grounding in Gemini Apps and Vertex AI.
Which AI crawlers should a small business allow?
Allow the AI search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Keep Googlebot and Bingbot open. Most small businesses with marketing pages can allow training bots too, but that is a choice about content reuse, not visibility.
Can robots.txt stop all AI bots from reading my site?
No. Robots.txt is a request, not a lock. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Fake bots can also spoof names. Private content needs logins or server-level blocking.
How long does a robots.txt change take to work for ChatGPT?
OpenAI says ChatGPT search takes about 24 hours to adjust after a robots.txt update. Answers can lag longer, because a page must be fetched again before it can be cited. Recheck your logs after a day, then test prompts a week or two later.
Why is my site on Google but not in ChatGPT or Perplexity?
Check access first. A wildcard Disallow, a firewall AI bot setting or a staging robots.txt can block AI crawlers while Googlebot still gets in. If access is fine, the gap is usually content clarity, entity details or weak third-party mentions.
Do I need a separate robots.txt for my subdomains?
Yes. Robots.txt works per host, so blog.example.com and shop.example.com each need their own file at the root. Copy the same AI crawler decisions into each one, or a subdomain may block AI search bots without anyone noticing.
Related reading
Not sure AI tools can see your site? Get a technical check from AllureSEO
Send us your domain and the AI tools your buyers use. We will check your robots.txt, firewall responses and server logs, then tell you plainly what is blocking AI crawlers and what to change. Reach us through our contact page, or call (718) 252-5040 or (212) 301-7615.