- Crawling, indexing and ranking remain the foundation of search, including for AI Overviews and AI Mode. In Google's words, optimizing for AI search is still SEO.
- Googlebot only processes the first 2 MB of an HTML file, so key content, canonicals and structured data belong near the top.
- The Google-Extended robots.txt token controls Gemini training, not inclusion in Google Search. OpenAI lets you control its training and search crawlers separately.
- Content that only renders via JavaScript in the browser is a risk. Server-rendered pages are fully visible to every crawler.
Search engines stopped being simple directories a long time ago. With AI Overviews and AI Mode, Google serves ready-made answers, while ChatGPT and Perplexity respond to questions with recommendations and sources. So many people are asking whether the fundamentals of search engine optimization still apply.
They do. Google's guide to AI search states explicitly that optimizing for generative search is still SEO. An AI answer can only cite content that has already been crawled, indexed and judged trustworthy. This guide explains how those three steps work, how to control AI crawlers and what matters for AI search.
Crawling is the prerequisite for any visibility
During crawling, programs such as Googlebot fetch pages to discover content. Google does not crawl everything it finds. Resources are limited, so a website has to make the crawler's job easy: fast responses, clear structure and few unnecessary URLs.
How Googlebot works
Googlebot is a globally distributed system that fetches pages much like a browser. Because Google indexes the mobile version of most websites, most requests come from Googlebot Smartphone.
How it works, in three steps:
- Discover: Google finds URLs through links, sitemaps and Search Console submissions.
- Queue: Discovered URLs go into a queue.
- Schedule: A scheduler decides when each URL gets fetched, based on crawl budget.
Googlebot only processes the first 2 MB (uncompressed) of an HTML file and the first 64 MB of a PDF. Anything beyond that is not indexed. That is why key content, canonical tags and structured data belong near the top of the code.
On large websites, crawl budget decides what gets indexed
Crawl budget rarely matters for small websites. For large online stores and news portals, it is critical. It comes down to two factors.
Crawl capacity: how much can your server handle?
- It depends on the health of your server.
- Fast response times lead to more crawling.
- Server errors (5xx) make Google throttle crawling immediately.
Crawl demand: how much does Google want to crawl?
- Popularity: Pages with many links pointing to them get crawled more often.
- Freshness: Google revisits frequently changing pages, such as a news home page, more often than static pages such as the legal notice.
Every low-value URL eats up budget your important pages need:
| Factor | Mechanism | Effect |
|---|---|---|
| Faceted navigation | Filters for color, size and price generate a nearly infinite number of URL combinations. | Crawlers get trapped and fetch millions of near-identical pages. |
| Soft 404 errors | The page says “Not found” but returns a 200 (OK) status. | Google wastes crawls on error pages. |
| Duplicate content | The same content under different URLs, for example because of tracking parameters. | Less capacity left for unique content. |
| Low-quality content | Mass-produced copy or thin pages. | Google may crawl the entire site less often. |
Our article on the XML sitemap in the age of AI search explains how a sitemap helps Google prioritize.
robots.txt lets you control search engines and AI crawlers
robots.txt is the house rules for crawlers. Unlike a firewall, it does not technically block anything, but reputable crawlers follow its instructions. Keep in mind: a robots.txt block prevents crawling, not indexing. To keep a page out of the index, use noindex and leave the page crawlable.
For AI crawlers, let your business model decide
Besides search engines, AI providers also crawl websites, both to train their models and to power search in their assistants. The major providers now keep these purposes separate:
- OpenAI: GPTBot collects training data, OAI-SearchBot finds sources for ChatGPT search, and ChatGPT-User fetches pages when a user asks for them in a conversation.
- Anthropic: ClaudeBot for training, plus separate agents for search and user-initiated retrieval.
- Google: The Google-Extended robots.txt token controls whether content is used to train future Gemini models and to ground answers in Gemini and Vertex AI. According to Google, it has no effect on inclusion or ranking in Google Search. Whether your pages appear in AI Overviews and AI Mode is controlled through Googlebot, snippet settings such as nosnippet, and a dedicated setting in Search Console. Opt out there, and you will no longer get impressions or traffic from these features.
The right call depends on your business model:
| Type of website | Recommendation | Rationale |
|---|---|---|
| Online stores and B2B companies | Allow search crawlers | You want your products and brand mentioned and recommended in AI answers. |
| Publishers and news sites | Weigh the trade-offs | Content is the core asset. Blocking training crawlers can strengthen your hand in licensing negotiations, but it costs visibility if search crawlers are blocked too. |
| Paid content | Block training crawlers | Content customers pay for should not become freely available through AI answers. |
Before blocking any crawler, check what it is for. Block a provider's search crawler, and you disappear from its answers.
HTTP status codes tell Google how to handle a URL
Every server response contains a status code. The HTTP status code tells Google whether a page is reachable, redirects or no longer exists.
- 200 (OK): The page is reachable and can be indexed.
- 301 (Moved Permanently): The most important code for site migrations. Google passes the old URL's signals on to the new one.
- 304 (Not Modified): The content has not changed since the last crawl, which saves crawl capacity.
- 404 (Not Found) and 410 (Gone): The content no longer exists. Google treats both codes almost identically and gradually drops the URL from the index. 404 errors are a normal part of the web, as long as no important links point to them.
- 5xx (server error): Critical. Google throttles crawling, and if the errors persist, pages drop out of the index.
Server-rendered content is visible to every crawler
Many websites build their content with JavaScript frameworks such as React, Vue or Angular. Google has to execute (render) these pages like a browser to see the content. That takes time and can fail.
Your architecture determines indexability
Client-side rendering (CSR): The browser or crawler has to assemble the page itself.
- Risk: If rendering fails or takes too long, the crawler sees a blank page. Many AI crawlers do not run JavaScript at all and never see this content.
Server-side rendering (SSR) and static pages: The server delivers finished HTML.
- Advantage: Every crawler gets the full content right away. For visibility in search and AI answers, this is the safest choice.
Hydration: The server delivers visible HTML, then JavaScript makes it interactive.
- Watch out: If hydration takes too long, the page is visible but unresponsive. That hurts your Core Web Vitals, especially INP.
Our article on the JavaScript problem of AI crawlers explains why this matters so much for AI search.
Google only indexes what it considers valuable
Google does not index everything it crawls. Indexing is a selection based on quality and usefulness.
Two Search Console statuses show you where to start
In the “Page indexing” report, two reasons regularly cause confusion:
- “Crawled – currently not indexed”: Google has fetched the page but has not (yet) indexed it, often because it is too thin or too similar to other pages. The fix is usually to improve the content or consolidate duplicates.
- “Discovered – currently not indexed”: Google knows the URL but has not fetched it yet, often because crawl capacity falls short. The fix: faster servers, fewer unnecessary URLs and stronger internal linking.
E-E-A-T is the foundation of quality assessment
Especially for health, finance and safety topics (YMYL), Google looks at whether content demonstrates experience, expertise, authoritativeness and trustworthiness:
- Experience: evidence of real-world use, such as your own photos, tests and figures.
- Expertise: named authors with relevant credentials and a profile.
- Authoritativeness: links and mentions on respected websites.
- Trustworthiness: security, transparency and accessible customer service.
In AI search, original content wins
According to Google, AI Overviews and AI Mode rely on the same ranking and quality systems as traditional search. The prerequisite is that a page is indexed and eligible to show a snippet. Google says you do not need special structured data, an llms.txt file or content chopped into small chunks.
What counts is original content:
- A point of view: Google recommends content with a distinct perspective and first-hand experience. A rehash of what is already everywhere gives a system no reason to cite you in particular.
- Clear answers: Answer each section's question up front, then explain your reasoning. This helps readers and systems alike.
- A clearly identifiable brand: Structured data remains useful for rich results and helps Google identify your company unambiguously.
Our article on generative engine optimization explains how language models choose their sources beyond Google.
Example: Organization markup (JSON-LD)
This markup helps Google correctly match your company's name, website and profiles:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Beispiel GmbH",
"url": "https://www.beispiel.de",
"logo": "https://www.beispiel.de/logo.png",
"sameAs": [
"https://www.linkedin.com/company/beispiel",
"https://www.youtube.com/@beispiel"
]
}
</script>
Conclusion: clean tech and original content make you visible in every kind of search
Search success rests on three foundations that depend on each other:
- Technology: fast servers, server-rendered content, strong Core Web Vitals and a crawl budget that goes to your important pages.
- Content: copy grounded in your own experience and data that adds something people cannot find elsewhere.
- Clarity: a clearly defined brand, clean structured data and a deliberate decision about which crawlers may use your content.
Remove the technical hurdles and establish yourself as a trustworthy source, and you stay visible in traditional search and AI answers alike. Our SEO and GEO pages show how we build websites for both.
