No. Blocking every AI crawler in robots.txt is a common default, but it can quietly remove a firm from AI search answers in tools like ChatGPT search and Google AI Overviews. The fix isn’t “allow everything” or “block everything” — it’s knowing which bot does what, because a training crawler and a search crawler are not the same thing.
What’s the difference between a training crawler and a search crawler?
AI companies run separate crawlers for separate jobs, and they are controlled separately in robots.txt. OpenAI’s own crawler documentation says it plainly: OpenAI uses OAI-SearchBot to surface a site in ChatGPT’s search results, and a completely different bot, GPTBot, to crawl content that may be used to train its generative AI models. A webmaster can allow one and disallow the other — “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training,” per the same documentation.
Other AI companies follow the same split. Google lists Google-Extended as a separate, named crawler from Googlebot in its own crawler documentation — Googlebot handles ordinary search indexing (including what feeds AI Overviews), while Google-Extended is the control specifically tied to Gemini model training. Perplexity and others also run distinct bots for live-answer retrieval versus any model-training use. The practical point for a law firm: a robots.txt rule written for one purpose (“keep our content out of AI training data”) can accidentally take out a different purpose (“show up when someone asks an AI assistant about our practice area”) if it’s written as one blanket “disallow all AI bots” rule instead of naming bots individually.
What happens if a firm blocks every AI crawler?
It opts out of both uses at once — training and the AI-answer features that rely on an allowed search/retrieval crawler. If a firm disallows OAI-SearchBot specifically, OpenAI’s documentation confirms the consequence directly: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” A copy-pasted “block all bots with AI in the name” snippet usually disallows that search bot along with the training bots, even though the firm may have only meant to opt out of training.
There’s also a lag to be aware of either way: OpenAI notes it can take roughly 24 hours after a robots.txt update for its systems to adjust, so a change doesn’t take effect immediately in either direction.
What can robots.txt actually control — and what can’t it?
This is the part firms get wrong most often, and it matters for anyone tempted to treat robots.txt as a privacy or security tool. Google’s own developer documentation is explicit: a robots.txt file “is used to manage crawler traffic to your site… it is not a mechanism for keeping a web page out of Google,” and more broadly, “the instructions in robots.txt files cannot enforce crawler behavior on your site; it’s up to the crawler to obey them.” Google even recommends that if you want information genuinely secure from crawlers, use password-protection instead, because other blocking methods actually restrict access.
In plain terms: robots.txt is a note left at the front gate asking visitors to stay off certain paths. Reputable crawlers from major AI companies generally read and follow it. But nothing about the file prevents a page from being fetched, and it was never designed to protect confidential information. A law firm should never rely on robots.txt to keep client data, intake forms, or internal pages private — that’s a job for authentication, password protection, or simply not publishing the page at a public URL.
What should a law firm’s robots.txt actually say?
A conservative, sensible default looks less like “block AI” and more like “be specific about which bot and which use”:
- Allow ordinary search crawlers (Googlebot, Bingbot) so the site keeps ranking normally.
- Decide deliberately on AI search/retrieval bots (such as OAI-SearchBot) based on whether the firm wants to be eligible for citation in AI-generated answers — for most firms trying to be found, allowing these makes sense.
- Decide separately, and explicitly, on AI training bots (such as GPTBot or Google-Extended) based on the firm’s own comfort with its public content being used in model training — this is a business judgment call, not a security control.
- Keep genuinely sensitive pages (client portals, intake forms, internal tools) off robots.txt entirely and behind real authentication, since robots.txt cannot enforce privacy.
- Re-check the file after any website redesign or migration, since default templates often ship with an overly broad “disallow all” or an outdated bot list.
The right setting is a judgment call for each firm, not a universal rule — but it should be a deliberate choice between named bots, not one catch-all line.
Does blocking GPTBot hurt my firm’s Google ranking?
No. GPTBot and Googlebot are unrelated crawlers run by different companies for different purposes, so a GPTBot rule has no bearing on standard Google search rankings.
If I block an AI crawler, is my site’s content actually safe from AI training?
Only to the extent that the crawler’s operator honors the request — robots.txt is a voluntary signal, not an access-control mechanism, so it stops cooperative bots and nothing else.
Do I need to update robots.txt every time a new AI bot shows up?
Periodically, yes. New crawlers get introduced, and a robots.txt file written a year or two ago may not mention the bots that matter today — review it as part of any site update.
Not sure what your current robots.txt is doing — or whether your site is even showing up in AI search answers right now? Get a free AI Search Analysis at growthagents.ai/sitescan to see where your firm stands.
This post is informational and does not constitute legal advice.




