Chapter 4 of 5 · AI and SEO — Using AI Well, and Understanding AI Search

AI search: AI Overviews, ChatGPT, AI crawlers, and what you can control

No special files or markup are needed for AI Overviews; the new Search Console switch from 31 August 2026; what each OpenAI, Anthropic and Perplexity bot does; how to appear in AI search without being used for training; and the truth about llms.txt.

The first three chapters were about writing with AI. This one looks at the other side: AI that reads your site to answer people’s questions — Google’s AI Overviews and AI Mode, and assistants such as ChatGPT, Claude and Perplexity.

4.1 AI Overviews and AI Mode use the same index as Google Search

Google says no special optimizations are needed to appear in AI Overviews or AI Mode. The basic conditions are that a page is indexed and eligible to show with a snippet — everything the first series covered — and that the site has the setting in 4.2 switched on. These are conditions, not a guarantee: Google says indexing and serving are not guaranteed.

What Google says you do not need:

new machine-readable files, AI text files or Markdown;
special schema.org structured data;
content chopped into small chunks for AI.

All three statements are about Google Search only, and structured data still has its other uses (chapter 4 of the first series); it is just not required for the AI features. If someone sells “SEO for AI” built on these three things, ask which official document they are relying on.

4.2 The new Search Console switch (31 August 2026)

Since 31 August 2026, Search Console has a setting called Search generative AI control for every site worldwide:

Option	Effect
Include (the default for every site)	The site can appear as links in AI Overviews, AI Mode and AI features in Google Discover, and gets impressions and traffic from them
Exclude	The site does not appear in those features and gets no traffic or impressions from them at all

Worth knowing:

the setting is not used as a ranking signal for the rest of Search, so excluding does not lower your normal rankings;
it has nothing to do with training models — for that use Google-Extended (4.4);
Google says a change generally takes a few days, longer for some content.

Google’s documents still disagree slightly. The AI features page, last updated at the end of 2025, still says there are no additional requirements; the newer guide (July 2026) says a site must have this setting on to be eligible. Most sites need do nothing, because the default is on.

4.3 Page-level controls

To limit individual pages rather than the whole site, the existing directives work:

Directive	Effect on Google’s AI features
`nosnippet`	No snippet, and the content cannot be used as a direct input for AI Overviews and AI Mode
`max-snippet`	Limits how much of the content may be used
`data-nosnippet`	Excludes part of a page (on `div`, `span` or `section`, as in chapter 3 of the first series)
`noindex`	Not in Google Search at all, including the AI features

4.4 Google-Extended: about training, not search

Google-Extended is a name used only in robots.txt; it has no user agent of its own. It controls whether content Google crawls may be used to train future Gemini models and for grounding answers in Gemini Apps and Vertex AI.

Google states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. The Search Console help page adds that it is also how you limit training of the models behind AI responses in Search.

User-agent: Google-Extended
Disallow: /

4.5 Other AI companies’ bots: know which does what

Most AI companies run several bots with separate jobs; blocking one does not block the others.

Company	Search bot	Training bot	User-initiated fetcher
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User
Perplexity	PerplexityBot (not used for training)	—	Perplexity-User
Common Crawl	—	CCBot (an open archive widely used to train models)	—

A common wish is to appear in ChatGPT search but not be used for training. OpenAI says you can, because each token is set independently:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

A caution when adding this to an existing file. Under the robots.txt standard, a bot with its own User-agent group follows only that group and ignores the User-agent: * group. If your file has a * group disallowing paths such as /admin/, copy those Disallow lines into the OAI-SearchBot group too, or leave the OAI-SearchBot group out altogether: with no rule against it, it may crawl anyway.

OpenAI says robots.txt changes take about 24 hours to reach ChatGPT search; Perplexity says up to 24 hours.

4.6 robots.txt is not a lock

The robots.txt standard (RFC 9309) describes its rules as ones crawlers are “requested to honor”, and states they are “not a form of access authorization”. Listing a path in the file also tells everyone that it exists.

Fetchers started by a user differ from crawlers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, since a user asked for the page. Anthropic says its bots honour robots.txt.

If content must truly stay private, use a password or login, not robots.txt.

4.7 llms.txt: the facts

llms.txt is a proposal by Jeremy Howard, published on 3 September 2024: a Markdown file summarising a site and linking its key pages for AI to read when answering. The proposal’s own page calls itself an “informal overview”; it has not gone through any standards body.

What can be checked:

Google states that Google Search does not use llms.txt; having one neither helps nor harms.
We found no official statement from OpenAI, Anthropic or Perplexity that their bots read llms.txt on other people’s sites.
The proposal was revised to version 2 in August 2026, and some companies, such as OpenAI and Perplexity, publish an llms.txt for their own developer documentation — which is not the same as their bots reading other sites’ files.

A lesson from KoishiAI: decide knowing what you are choosing

This site lets every AI bot in, including training crawlers, because we want our content cited by AI assistants, and we publish an llms.txt.

But we do not believe llms.txt helps with Google. It is simply a description of the site, kept in case other systems use it, at almost no cost. Allowing training is a business decision, not a technical one; a site that sells content or holds unique data may choose differently, and is not wrong to.

4.8 Match the setting to the goal

You want	Do this
To appear in AI Overviews and AI Mode	Normal SEO, pages indexed, setting left on “include”
Google not to train Gemini on you, while staying in Search	Block Google-Extended in robots.txt
Out of Google’s AI features (including AI features in Discover), but still in normal results	Set Search generative AI control to “exclude”
In ChatGPT search, but not used for training	Allow OAI-SearchBot, block GPTBot
Anthropic not to train on your future content	Block ClaudeBot
Nobody to see it	Use a login

4.9 After this chapter

Open Search Console and check what your Search generative AI control is set to, and whether that is what you want.
Read your robots.txt and say, for each bot in table 4.5, whether it may enter, and whether you meant that.
If someone offers to sell you llms.txt or special markup for AI Overviews, ask for the official documentation behind it.

The last chapter covers measuring visits from AI search, and the risks to watch.

AI crawler checker — paste a robots.txt and see which bots may read which pages, and generate rules