- Google calls the sitemap a hint, not a guarantee. It only works if it contains exclusively canonical, indexable pages.
- A sitemap file holds a maximum of 50,000 URLs or 50 MB uncompressed. We split large catalogs by category so that gaps in indexing can be quickly assigned.
- Google ignores priority and changefreq. Google only uses lastmod if the date consistently matches the last significant change.
- According to Google, no special rules apply to AI Overviews and AI Mode: a page must be indexed. An llms.txt is not needed for this.
If a new product doesn't appear on Google, we check the XML sitemap first. It is the list of addresses a store suggests to the search engine for indexing.
According to Google, a submitted sitemap is only a hint. Google decides itself whether to fetch it and which addresses to crawl. That makes it all the more important that every entry is correct. In this article, we show how we check and organize an online store's sitemap.
We start every audit with submitted and indexed URLs
In Google Search Console, the “Pages” report can be filtered by individual sitemaps. It shows how many of the submitted addresses are in the index and why the rest are missing.
If the sitemap is split by category, the report also shows where the gap is. If, for example, most addresses are missing in the “Shoes” sitemap, the search starts there and not across the entire store.
Google names the reasons in the report itself, including redirects, pages with noindex, duplicates without their own canonical address, and pages that Google has crawled but not indexed. Each reason requires a different fix.
How the other Search Console reports help is described in our article on Google Search Console.
Only pages that should appear on Google belong in the sitemap
A sitemap is not a list of all of the store's addresses. It contains only pages that respond with status 200, are canonical, and may be indexed. This is how we classify the typical page types of a store:
| Page type | In the sitemap? | What else we do |
|---|---|---|
| Product page with its own address | yes | Set lastmod when content, price, or availability changes |
| Variant without its own search demand (color, size) | no | Canonical to the product's main address |
| Variant with its own demand and its own content | yes | Separate page with its own text |
| Sorting and filters without their own search demand | no | Exclusion via robots.txt or canonical |
| Temporarily sold-out product | yes | Show availability on the page and in the structured data |
| Permanently removed product | no | 301 to an equivalent successor product, otherwise 404 or 410 |
| Page with noindex | no | Remove from the sitemap at the same time |
| Address with tracking or session parameters | no | Set internal links without such parameters |
We specifically check two contradictions: pages with noindexthat are still in the sitemap, and addresses in the sitemap that the robots.txt blocks. In both cases, Search Console reports errors, and the sitemap loses its meaningfulness.
What a cleaned-up structure can achieve is shown by Shirtinator. There, we cleaned up pages that were no longer accessible and excluded sorting pages via the robots.txt. The crawl rate rose by 19%, organic visibility by 28%.
lastmod only works if the date is correct
Google ignores the fields priority and changefreq. Google uses the field lastmod if it consistently and verifiably matches the last significant change. According to Google, a new year in the footer doesn't count.
Some store systems set the current date on every request. Then the field no longer carries any information. We check this with samples: products that have been unchanged for months must not carry a fresh date.
The former ping to Google no longer exists. Google deprecated the endpoint in June 2023. You submit new sitemaps via Search Console, the Search Console API, or a line Sitemap: in the robots.txt . Google learns about changes through a correct lastmod.
We split large catalogs before the limit is reached
A sitemap file holds a maximum of 50,000 URLs or 50 MB uncompressed. Larger catalogs need a sitemap index, i.e., a file that points to several sitemaps.
We split according to the store's structure:
- by category or brand, so that gaps in indexing can be assigned to a product group;
- by content type, i.e., products, categories, guides, and static pages separately;
- by country or language for international stores.
The store systems' plugins provide a basis for this. If it isn't enough, the sitemap is generated directly from the product database. We describe this task for your IT team or implement it ourselves if we have access to the system.
International stores assign language versions in the sitemap
If you sell in several countries, hreflangtells Google which page applies to which country. The information can be in the HTML head, in the HTTP header, or in the sitemap. For large catalogs, we maintain it in the sitemap because it is all in one place there.
<url>
<loc>https://example.com/de-at/produkt</loc>
<xhtml:link rel="alternate" hreflang="de-at" href="https://example.com/de-at/produkt"/>
<xhtml:link rel="alternate" hreflang="de-de" href="https://example.com/de-de/produkt"/>
</url>
Each language version refers to all the others and to itself. If a return reference is missing, Google cannot use the assignment.
How much depends on this is shown by Gradient Zero. For the international website, impressions in search rose by 123% in six months. More about our work for international websites is on the SEO International page.
The same index counts for Google's AI search
According to Google, optimizing for AI Overviews and AI Mode is still SEO. A page must be indexed and eligible to appear with a snippet in search. According to Google, you don't need an llms.txt or special structured data for this.
For your sitemap, this means: it helps Google find new and changed products. Only then can these products appear in an AI answer from Google.
ChatGPT, Perplexity, and other AI services use their own crawlers. You determine whether they may fetch your pages in the robots.txt . How we get businesses into AI answers is described on the page GEO.
We check the sitemap after every major change to the store
Product range changes, migrations, and new plugins change a store's addresses. That is why we check the sitemap after such changes and compare it with a crawl of the store. This way we find entries that no longer exist and pages that aren't in any sitemap.
For us, this check is part of technical SEO. There, we first fix the errors that affect your highest-revenue pages.
