What each crawler controls
| Crawler | What it is for | If you block it |
|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | Your pages drop out of Google Search |
| Google-Extended | Described by Google as covering AI uses in some of its other systems | Google lists Googlebot rules and snippet controls, not this, as its Search controls |
| OAI-SearchBot | ChatGPT search | Not shown in ChatGPT search answers |
| GPTBot | OpenAI model training | Signals content should not be used for training |
| ChatGPT-User | Actions a person asks ChatGPT to take | OpenAI notes robots.txt rules may not apply |
How to check your site
Read your robots.txt, then check your hosting or security service, since some offer settings that block AI crawlers on their own. OpenAI says robots.txt changes can take about a day to affect ChatGPT search.
Go deeper
The AI SEO guide for lawyers covers crawlers alongside the rest of the technical foundation. If you are weighing an llms.txt file at the same time, see whether a firm needs one.