What Google and Microsoft say about AI Overviews, AI Mode and Copilot, what still decides whether pages are found and cited, what to ignore, crawler controls, measurement and what we changed on our own site.
Search now answers as often as it lists. Google shows AI Overviews and offers AI Mode, ChatGPT and Perplexity search the web and cite what they find, and Microsoft’s Copilot draws on Bing’s index. Business owners ask us two questions about this: does SEO still matter, and what should we do differently to be the source these answers cite?
This guide answers both, using what the search companies say themselves rather than guesses. It covers what changed between 2024 and 2026, what still decides whether your pages are found and cited, what you can safely ignore, how to control the way AI systems use your content, how to measure it, and what we changed on our own site. The detailed, page-level checklist for getting cited sits in our companion guide, how to get cited in AI Overviews, ChatGPT and Perplexity.
Every rule and date below links to its official source, each checked on 13 Nov 2026. Search platforms change their guidance often, so we review this article every quarter.
Key takeaways
- Google states there are no extra technical requirements and no special markup to appear in AI Overviews or AI Mode: a page must be indexed and eligible to show a snippet.
- AI answers are built by retrieving pages from a search index, so crawlable, indexable, genuinely useful pages are still the foundation of being cited.
- Google says it does not use llms.txt files, "chunked" content or AI-specific rewriting, and that seeking inauthentic mentions does not help.
- You can control AI use separately from search: robots.txt tokens for ChatGPT, Perplexity and Google's Gemini training, and new Microsoft AI preferences from February 2027.
- Measure AI visibility with Search Console's Generative AI performance report and referral traffic, not with tools that claim access to Google's internal metrics.
What changed in search between 2024 and 2026
The basics of crawling and indexing have not changed. What changed is how results are presented and which crawlers visit your site. The milestones below are the ones that affect a business site directly.
3 Sep 2024
The llms.txt proposal
Jeremy Howard publishes a proposal for a /llms.txt file to help AI agents read a website. Version 2 follows on 10 Aug 2026.
7 May 2026
FAQ rich results end
Google stops showing the FAQ rich result in Search. The FAQPage markup remains valid schema.org, but no longer earns an enhanced listing on Google.
15 May 2026
Google publishes an AI optimisation guide
Google's new guide to generative AI features in Search says optimising for AI search is still SEO and lists "AEO/GEO" tactics site owners can ignore. On 15 Jun 2026 it adds that Google Search does not use llms.txt files.
30 Oct 2026
Microsoft announces AI preferences
Bing introduces separate AI Training and AI Grounding preferences that will replace NoArchive and NoCache for these uses.
Early 2027
Shared training signal
Google says it intends to support the shared AIPREF training preference signal in robots.txt in early 2027.
1 Feb 2027
Microsoft starts honouring the new preferences
Bing begins reading AI Training and AI Grounding preferences as it crawls; NoArchive and NoCache stop counting for training and grounding after 1 Mar 2027.
How an AI answer is built from your pages
It helps to know what happens between a question and an AI answer, because it explains why classic SEO still matters. Google’s own description of AI Overviews and AI Mode, in its guide to optimising for generative AI features, names two techniques:
Step 1: The question is expanded
With "query fan-out", the model issues several related searches across subtopics. Google's example: a question about a lawn full of weeds also searches for herbicides, chemical-free removal and prevention.
Step 2: Pages are retrieved from the index
The core Search ranking systems retrieve relevant, up-to-date pages from the Search index. Google calls this retrieval-augmented generation, or grounding.
Step 3: The answer is written and linked
The model reviews the retrieved pages, writes a response and shows prominent links to the pages that support it.
The consequence is simple: a page that is not crawled, indexed and judged relevant by the normal ranking systems is not available to be cited. ChatGPT search and Perplexity work on the same principle with their own crawlers and indexes, which is why their documentation asks site owners to let those crawlers in. If you want the technical detail of retrieval, our explainer on retrieval-augmented generation covers how it works inside any AI system.
What Google and Microsoft say themselves
A lot of advice about “AI search optimisation” is speculation. The official statements are short and consistent. Google’s AI features and your website page is the place to start:
There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.
The same page sets the technical bar: to be shown as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet. Google’s newer optimisation guide adds that a site must be included in the “Search generative AI features” setting in Search Console, and it answers the AEO question directly:
From Google Search's perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.
Microsoft’s guidance for Bing is about control rather than ranking. Its webmaster guidelines recommend IndexNow so Bing hears about new, changed and deleted URLs quickly, which reduces outdated references in Copilot answers. Its 30 Oct 2026 announcement introduces separate preferences for AI training and AI grounding, covered below.
Classic SEO and AI answers compared
| Question | Classic results | AI answers (AI Overviews, AI Mode, ChatGPT, Perplexity) |
|---|---|---|
| How content is found | Crawled and indexed by the search engine | The same: retrieved from a search index built by crawlers |
| What success looks like | A ranking position and a click | A citation or supporting link inside the answer, and a click from it |
| Eligibility on Google | Indexed and policy compliant | Indexed and eligible to show a snippet; no extra requirements |
| Special markup needed | Structured data for rich results | None; Google says no special schema.org markup is needed |
| How many sources are shown | Ten links per page | Several supporting links, often from more diverse sites because of query fan-out |
| Where to measure on Google | Search Console Performance report | The same report (Web search type) plus the Generative AI performance report |
| Crawler controls | robots.txt and robots meta tags | The same for Google Search, plus separate tokens for AI training and other AI products |
Source: Google Search Central: AI features and your website; Optimizing for generative AI features; OpenAI and Perplexity crawler documentation; checked 13 Nov 2026
What still decides whether you are found and cited
Technical access: let the right crawlers in
Google’s list of SEO fundamentals for AI features starts with crawling: allow it in robots.txt and in any CDN or hosting firewall. It is easy to miss: security rules and bot filters on CDNs can block AI search crawlers without anyone noticing. Perplexity’s crawler documentation gives firewall instructions for exactly this reason, and OpenAI recommends allowing OAI-SearchBot and requests from its published IP ranges if you want to appear in ChatGPT search.
Beyond access, the usual technical work applies: pages that return the right status codes, canonical URLs, internal links that make content easy to find, important content in text rather than only in images, and a page experience that works on every device. Our SEO services cover this audit in full.
Content that is worth citing
Google’s guide is direct about what makes content stand out in AI search: “non-commodity” content that offers a unique point of view or first-hand experience, rather than a summary of what is already online. Its example contrasts “7 Tips for First-Time Homebuyers” with an article explaining why one buyer waived an inspection and saved money. For a business, non-commodity content usually means:
- your own prices, process, timelines and decisions, stated plainly;
- examples from your real work, named where the client agrees;
- comparisons and data with their sources and dates;
- answers to the questions your sales team hears every week.
Google’s helpful content guidance adds a useful test it calls “Who, How, and Why”: is it clear who wrote the content, how it was produced, and why it exists? Bylines that lead to an author page with real background are part of that “Who”.
Consistent facts about your business
AI answers about local services and products draw on structured sources as well as web pages. Google’s guide names Merchant Center product feeds and Google Business Profiles as ways to keep your products and services visible in AI responses. If your opening hours, prices or service areas differ between your website, your Business Profile and your feeds, an AI answer may repeat the wrong one. Our local SEO work starts with making those match.
Structured data that matches the page
Structured data is not required for AI features, and Google says there is no special schema for them. It still helps search engines understand a page and remains the route to rich results. The rule Google repeats is that structured data must match the visible text. Article markup with a named author, an author URL and a dateModified is worth having on every article; Google’s Article structured data documentation recommends marking up an internal author page with profile page markup.
Want a second opinion on your plans?
Tell us what you are working on. A specialist will look at it and reply with practical next steps.
Get in touchWhat you can safely ignore
Google’s optimisation guide includes a “mythbusting” section. For Google Search, it says you can ignore:
- llms.txt and other “special” files or markup. Google Search does not use them, and they neither help nor harm rankings. It is fine to publish one for other services that read it.
- “Chunking” content into tiny pieces. Google’s systems understand pages that cover several topics, and there is no ideal page length.
- Rewriting content just for AI. AI systems understand synonyms, so you do not need every variation of a query on the page.
- Seeking inauthentic mentions. Buying or planting mentions across the web does not help; Google’s ranking and spam systems both apply to AI features.
- Overfocusing on structured data. Useful for rich results, not a requirement for AI answers.
The guide also warns that creating separate pages for every fan-out query, primarily to manipulate AI answers, violates Google’s scaled content abuse policy. The same policy names “using generative AI tools or other similar tools to generate many pages without adding value for users” as an example. A content plan built on volume is a risk, not a strategy.
Controlling how AI systems use your content
You can now separate “show my pages in search and AI answers” from “use my content to train models”. Each company uses its own crawler names and tokens.
| Company | Name or token | What it controls | Effect of blocking |
|---|---|---|---|
| Googlebot, plus nosnippet, data-nosnippet, max-snippet and noindex | Google Search, including AI Overviews and AI Mode | Blocking or noindex removes pages from Search itself; snippet controls limit what is shown | |
| Google-Extended (robots.txt token) | Training future Gemini models, and grounding in Gemini Apps and Vertex AI | No effect on Google Search inclusion or ranking | |
| OpenAI | OAI-SearchBot | Showing sites in ChatGPT search answers | Pages are not shown in ChatGPT search answers, though they can appear as navigational links |
| OpenAI | GPTBot | Use of crawled content to train OpenAI's foundation models | Content is marked as not to be used for training |
| Perplexity | PerplexityBot | Showing and linking sites in Perplexity search results; not used for training | Pages are left out of Perplexity's search results |
| Perplexity | Perplexity-User | Pages fetched when a user asks a question | Perplexity says this fetcher generally ignores robots.txt, as the user asked for the page |
| Microsoft | AI Training (robots.txt or page) and AI Grounding (page level) | Use of content for training and for grounding Copilot answers | Honoured from 1 Feb 2027; replaces NoArchive and NoCache for these uses |
Source: Google common crawlers documentation; Google AI features page; OpenAI crawlers documentation; Perplexity crawlers documentation; Microsoft Bing announcement of 30 Oct 2026; all checked 13 Nov 2026
A robots.txt file that stays visible in Google, ChatGPT and Perplexity search while opting out of model training looks like this. Treat it as an example to adapt, not a recommendation: whether to allow training is a business decision.
# Search engines and AI search: allowed
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Model training: opted out
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Shared training preference (Microsoft from Feb 2027; Google intends early 2027)
User-agent: __usage_ai-training
Disallow: /
Sitemap: https://www.example.com/sitemap_index.xmlMeasuring visibility in AI answers
Measurement is still young, and the honest position is that you will see part of the picture, not all of it.
- Search Console. Google counts AI Overviews and AI Mode in the overall Performance report under the “Web” search type, and offers a Generative AI performance report for its AI features in Search and Discover.
- Analytics. Visits from AI assistants that link to you arrive as referral traffic when the browser passes a referrer. Track them as their own channel, and track conversions, not only visits. Google states that clicks from results pages with AI Overviews are of higher quality, with people more likely to spend more time on the site.
- Manual checks. A fixed list of the 20 to 50 questions your buyers ask, run monthly in each AI search product, shows whether and how you are cited. Record the date, the answer and the sources.
Be careful with tools that promise AI rankings. Google’s guide says plainly that no third-party tool has access to its internal ranking or AI systems.
A step-by-step plan for 2026 and 2027
Step 1: Check crawler access
Review robots.txt, CDN bot rules and firewall settings for Googlebot, Bingbot, OAI-SearchBot and PerplexityBot. Confirm with server logs that each one reaches your key pages.
Step 2: Confirm indexing and eligibility
Use Search Console to check that important pages are indexed and can show snippets, and that the Search generative AI features setting includes your site.
Step 3: Fix your business facts
Make your website, Google Business Profile and product feeds agree on services, prices, hours and locations.
Step 4: Rewrite your top 20 pages with first-hand detail
Add your own process, prices, examples and dated sources. Remove generic sections that any competitor could have written.
Step 5: Show who wrote it
Add bylines that link to real author pages, with Article and profile page structured data that match the visible text.
Step 6: Decide your AI training policy
Choose which training crawlers to allow, set the robots.txt rules, and plan the Microsoft AI Training and AI Grounding preferences before 1 Mar 2027.
Step 7: Measure monthly
Track the Generative AI performance report, AI referral traffic and a fixed question list. Change one thing at a time.
Step 8: Earn real mentions
Publish useful research, talks and case studies that others cite on their own. Our digital PR work focuses on this kind of earned coverage.
One international site for eight markets
Many of our clients sell in several of the UK, USA, Canada, UAE, Australia, India, Singapore and Europe. The temptation is to create country copies of the same page, or a page per city with the place name swapped. Google’s spam policies name “doorway abuse”, including pages “targeted at specific regions or cities that funnel users to one page”, and AI fan-out makes them even less useful: one strong page that answers the question for every market is what gets retrieved.
What works instead is one page per topic, written in clear international English, with real differences shown where they exist: a table of what changes by market (law, payment methods, hosting location, time zones) and the countries you serve stated in your Organization and Service structured data. A separate country page makes sense only when its content is genuinely different, such as local law, a local team or local prices.
What we changed on our own site
We apply the same rules to acmasoft.com, and we describe them here because they are verifiable on the site:
- One international site. We serve buyers in eight markets from one English site with one URL per topic. We have no country or city copies, and so no hreflang tags.
- Markets in structured data. Our Organization and Service markup lists the eight markets we serve in
areaServed, rather than repeating country names in titles. - Real authors. Articles carry a byline from a verified staff profile, and author pages with a published bio are marked up as a profile page with a Person. Our article markup carries the author and the last reviewed date as
dateModified. - FAQs for readers. We keep short FAQ sections on articles and service pages because readers use them, and we keep the FAQPage markup because it matches the visible text, knowing Google no longer shows FAQ rich results.
- No llms.txt yet. Google says it does not use one. We have not published one, and we will revisit that when a service our buyers use documents that it reads them.
- Crawlers allowed. Our robots.txt blocks only the WordPress admin area, so search and AI search crawlers can read every public page.
Checklist: SEO for AI search
Quarterly SEO and AI search review
- Googlebot, Bingbot, OAI-SearchBot and PerplexityBot can reach key pages through robots.txt, CDN and firewall
- Key pages are indexed, can show snippets and are included in Search generative AI features
- Each important page answers its question with first-hand detail, sources and dates
- Bylines link to real author pages with background and credentials
- Structured data matches the visible text, including author and dateModified on articles
- Business facts match across the website, Business Profile and product feeds
- AI training preferences are set on purpose for Google, OpenAI and Microsoft
- IndexNow or sitemaps notify search engines of new, changed and deleted URLs
- No doorway pages, city copies or mass-produced pages targeting fan-out queries
- The Generative AI performance report, AI referrals and a fixed question list are reviewed monthly
Where AEO fits
“Answer engine optimisation” (AEO) and “generative engine optimisation” (GEO) are names for the AI-facing part of this work. Google’s position is that it is still SEO, and we agree on the substance: the same fundamentals decide both. In practice, our answer engine optimisation service adds the work that classic SEO projects often skip: the crawler and firewall audit for AI search bots, a monthly question list across AI products, entity and business facts that agree everywhere, and content rewritten around first-hand answers. Our SEO packages include the core technical and content work, and if you want people in your own team, you can hire SEO experts who work this way.
We will add guides on e-commerce SEO, technical SEO audits and local SEO for several locations to this series over the coming weeks.
Want to know how AI search sees your site?
- We audit crawler access, indexing, business facts and your top pages against the official guidance, then run your buyers' questions through Google, ChatGPT and Perplexity. See our SEO services to book an audit.
Get new guides by email
Articles and guides from our team, and company news. Unsubscribe at any time.
About AcmaSoft
AcmaSoft designs and builds websites, mobile apps and custom software, and runs SEO and digital marketing for its clients.
- Years of experience
- 17+
- Clients served
- 650+
- Projects delivered
- 2,000+
- Products shipped
- 300+


