The three kinds of AI crawler
| Job | Examples | What the company says |
|---|---|---|
| Search | OAI-SearchBot, PerplexityBot | OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Perplexity says PerplexityBot surfaces and links websites in its results. |
| Training | GPTBot, Google-Extended | OpenAI says disallowing GPTBot indicates content should not be used to train its foundation models. Google says Google-Extended does not affect inclusion or ranking in Google Search. |
| User request | ChatGPT-User, Perplexity-User | Both companies say these fetch pages because a person asked, and that robots.txt rules may not apply or are generally ignored. |
The details are in OpenAI's overview of its crawlers, Perplexity's crawler documentation, and Google's list of common crawlers. Google-Extended is not a separate crawler: Google describes it as a robots.txt token that controls whether content it crawls may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Googlebot itself serves Google Search, including AI Overviews and AI Mode.
What to check on your firm's site
Start with robots.txt, then check your hosting or security service, since some block automated traffic on their own. OpenAI's help center says that for a site to be eligible for ChatGPT search, the host or CDN must also allow traffic from OpenAI's published search crawler IP addresses. OpenAI and Perplexity both say robots.txt changes can take up to about 24 hours to reach their systems.
Whether to allow training crawlers is a policy choice for the firm, separate from search. The trade-offs are in whether a firm should block AI crawlers. Once access is settled, the next question is whether the pages are worth citing; see whether Perplexity cites law firm sites and how ChatGPT recommends lawyers.
Go deeper
The AI SEO guide for lawyers covers crawler access as part of the full technical foundation. Access only gets a page considered; the practice-area guides show what each practice's pages need to say once crawlers can reach them.
Common questions
Does blocking GPTBot keep my firm out of ChatGPT search?
Not by itself. OpenAI treats GPTBot and OAI-SearchBot as independent settings. Blocking GPTBot signals that your content should not be used for training; blocking OAI-SearchBot is what keeps a site out of ChatGPT search answers.
Keep reading
Sources
- OpenAI, "Overview of OpenAI crawlers" (linked above)
- OpenAI Help Center, "ChatGPT search"
- Perplexity, "Perplexity Crawlers" (linked above)
- Google Search Central, "Google's common crawlers" (linked above)