Should a law firm block AI crawlers?
Not with a blanket rule. A crawler is an automated program that visits web pages. Some AI crawlers collect content to train models, while others fetch pages so an AI product can show and link to them in answers. Blocking everything can remove your firm from AI answers. Decide on training and search separately, and on purpose.
Most firms we talk to never made this decision. A developer, a security plugin, or a hosting default made it for them. This post explains what each crawler does, what robots.txt can and cannot do, and how to check what your AI crawlers law firm website setup is actually doing today. It is marketing and technical information, not legal advice; your confidentiality obligations are governed by your state’s rules, so involve your ethics counsel for those questions.
What is the difference between training and search crawlers?
robots.txt is a small text file at the root of your site (yourfirm.com/robots.txt) that tells automated visitors which areas they are asked to stay out of. Each crawler identifies itself with a name, called a user agent. The key point is that one company often runs several crawlers with different jobs, and you can set rules for each one separately.
The table below reflects each vendor’s own documentation as of this writing. Names and behaviors change, so confirm against the linked pages before you edit anything.
| Crawler | Operator | Purpose, per vendor docs | What robots.txt does |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT’s search features | Sites opted out “will not be shown in ChatGPT search answers”; OpenAI says updates can take about 24 hours to take effect (OpenAI) |
GPTBot | OpenAI | Crawls content that may be used to train OpenAI’s generative AI foundation models | Disallowing it indicates your content should not be used for training (OpenAI) |
ChatGPT-User | OpenAI | Certain user-initiated actions in ChatGPT; not automatic crawling | OpenAI says robots.txt rules “may not apply” because a user initiated the action (OpenAI) |
ClaudeBot | Anthropic | Collects web content that could contribute to training | Blocking signals that your site’s future materials should be excluded from training (Anthropic) |
Claude-SearchBot and Claude-User | Anthropic | Search result quality; retrieval when a person asks Claude a question | Anthropic says blocking these may reduce visibility in user search results (Anthropic) |
PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results; not used for AI foundation model training | Perplexity recommends allowing it in robots.txt (Perplexity) |
Google-Extended | A robots.txt token controlling whether content Google crawls may be used for Gemini training and grounding | Does not affect inclusion in Google Search or ranking (Google) |
Two patterns matter for a partner or marketing director. First, the “training” bots (GPTBot, ClaudeBot, Google-Extended) are about whether your content feeds future models. Second, the “search” bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are what AI products use to find and link to pages when someone asks a question. OpenAI states the settings are independent, so a site can allow OAI-SearchBot and disallow GPTBot.
What does Google-Extended actually control?
Google describes Google-Extended as “a standalone product token that web publishers can use to manage” whether content Google crawls may be used for training future Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google’s cloud AI platform). Per Google’s documentation, it has no separate HTTP user agent string, so you will not see it as its own line in server logs.
Google also states plainly that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” That page does not mention AI Overviews, so we do not make claims here about how this token relates to them. If AI Overviews matter to your firm, read Google’s current Search documentation directly, and see our analysis of how Google AI Mode changed citation links for what we have measured.
What can robots.txt not do?
Four limits are worth knowing before you rely on it.
- It is a request, not enforcement. Vendors say their crawlers honor it, for example Anthropic states its bots respect “do not crawl” signals in robots.txt. Nothing in the file technically stops a visitor who ignores it.
- User-initiated fetches may not follow it. OpenAI says robots.txt rules “may not apply” to
ChatGPT-User. Perplexity says itsPerplexity-Userfetcher “generally ignores robots.txt rules” because a person requested the visit (Perplexity). Anthropic takes a different approach and documents that disablingClaude-Userprevents retrieval in response to a user query. Vendors differ, and they change. - It does not remove what was already collected. Anthropic’s documentation describes blocking as applying to future materials.
- It is public. Anyone can read your robots.txt, so listing sensitive paths in it advertises them.
What is a safe default for a law firm site?
There is no universal right answer, but a sensible starting position looks like this.
- Allow the search crawlers (
OAI-SearchBot,Claude-SearchBot,PerplexityBot) if you want any chance of being surfaced and linked in AI answers. No setting can guarantee a citation, but a blocked crawler cannot cite what it cannot read. - Decide on training deliberately. Some firms are comfortable with their public practice-area pages informing models. Others prefer to opt out. Either is defensible. The vendor documentation we reviewed does not frame training opt-outs as a ranking factor, so treat it as a policy choice, not a ranking tactic.
- Protect confidential areas with real access controls. Client portals, intake submissions, document-sharing links and internal tools belong behind logins, and ideally should not be reachable at all by anonymous visitors. Do not use robots.txt as your confidentiality control.
If your firm decides to allow search and opt out of training, an example file looks like this. Each directive follows the vendors’ documentation linked above; test it before deploying.
# Allow AI search crawlers User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Opt out of AI training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / If you would rather allow training too, simply remove the three opt-out groups. Remember that each subdomain (for example a separate blog host) needs its own file, per Anthropic’s guidance.
How do you check what your site is serving?
Many firms discover their rules do not match what they believed. Work through these three layers.
- Read your robots.txt. Open yourfirm.com/robots.txt in a browser. Look for a blanket
User-agent: *withDisallow: /, or an AI-bot list added by an SEO plugin or hosting dashboard that nobody remembers approving. Make sure it is the live file, not the one in your staging copy. - Check your CDN or firewall. A content delivery network (CDN) or web application firewall can block bots no matter what robots.txt says. Some security services have AI-bot blocking switches. Ask your developer or host whether any bot-management or “block AI scrapers” setting is on, and whether the search crawlers above are exempted if you want them allowed. Anthropic specifically advises against blocking its bots by IP address, because that may stop them from reading your robots.txt at all.
- Look at server logs. Your host’s access logs show which user agents requested which pages and what response they got. Seeing a crawler name with repeated 403 or 429 errors means something is blocking it. Seeing none is also informative, though absence does not prove a cause.
Crawl access is only one piece. Even a fully open site will not be cited unless the content itself is useful and clear, as we explain in why schema markup alone will not get you cited. Our methodology page describes how we measure citations.
What should you look for in a Best AI Search Platform?
If you are evaluating vendors, crawl-access checking is one criterion among several. When you compare a Best AI Search Platform, ask whether it:
- Checks whether AI crawlers can actually reach your pages, including robots.txt and bot blocks, not only your page copy.
- Measures how often your firm is cited in AI answers for the questions your clients ask, instead of reporting only rankings.
- Explains its method and its limits, and does not promise placement.
- Understands law firm realities such as confidentiality, practice-area pages and local intent.
For a sharper filter, see 7 questions to ask if your agency is selling rebranded SEO as GEO. For the broader picture of how firms are using AI in marketing, see our AI guide for law firms.
FAQ
Will blocking GPTBot remove my firm from ChatGPT?
Per OpenAI’s documentation, GPTBot relates to training, while OAI-SearchBot governs appearance in ChatGPT search answers. So blocking GPTBot alone is not described as removing you from search answers. Blocking OAI-SearchBot is. We cannot promise any outcome either way.
Is llms.txt a substitute for robots.txt?
No. An llms.txt file is a proposed convention for pointing AI tools to useful content, not an access-control standard, and the vendor documentation we reviewed for this post does not describe it as controlling crawler access. Robots.txt remains the file vendors document for crawler permissions.
Do I need my web developer?
For reading the file, no. For editing it, and especially for CDN or firewall settings, yes. Bring this post’s table and ask for the three checks above, then have them confirm in writing what is live.
Does blocking protect client confidentiality?
No. Robots.txt is a public, voluntary request about public pages. Confidential material should never be on a publicly reachable page, and should sit behind authentication. What your obligations require is a question for your ethics counsel, under your state’s rules.
See what AI crawlers can reach on your site
Want a plain-language read on whether AI systems can reach and use your pages? Run our Free AI Search Analysis. It is a readiness check, not a guarantee of citations, and it is a good place to start before you change a single directive.




