# Sitemap plus robots discovery pack

> Ship `/sitemap.xml` and `/robots.txt` together so crawlers can find the important routes fast instead of discovering the site only through navigation and luck.

- Canonical HTML: https://growth.iangoh.com/growth-ideas/sitemap-plus-robots-discovery-pack/
- Source: [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview)
- GrowthDex source hub: [Google Search Central](/sources/google-search-central-developers-google-com/)
- Last checked: 2026-05-28
- Rarity: common
- Budget: free
- Channels: SEO, AI Search
- Stages: crawl hygiene, technical seo, launch readiness, ai discovery


## Why this can grow

This works because discovery is often lost in the handoff between the site's real information architecture and the version a crawler can easily see. A sitemap names the pages you want indexed. Robots.txt gives access rules and can point straight to the sitemap. On content-heavy or JavaScript-heavy sites, that combination becomes the simple crawl map that keeps canonical pages easier to find.

## Ian's take

From scaling consumer platforms across MENA and Southeast Asia, my default is to distrust growth work that only looks good in a slide. For SEO and AI search, I care less about clever keyword tricks and more about clarity. A buyer, crawler, or answer engine should quickly understand who this is for, why it works, what proof backs it, and what page deserves to be cited. I would run it small enough to learn quickly, then only scale the parts that real users repeat, save, reply to, or buy from. For this tactic, I would watch one clear growth signal before putting more time or budget behind it.

## Action plan

1. Define one narrow startup segment where sitemap plus robots discovery pack can create a measurable lift.
2. Turn the tactic into one offer, page, campaign, or workflow for the SEO and AI Search channel.
3. Use the evidence from developers.google.com to set the first version of the message, format, and audience.
4. Launch a small test for 7 to 14 days with one success metric: one measurable growth signal.
5. Review the result, keep the winning message, remove weak variants, and turn the learning into a repeatable growth playbook.

## Source-backed example

Google Search Central describes sitemaps as a way to tell search engines which URLs on a site are available for crawling.

## Adjacent tactics in the same lane

- [Structured data as AI citation hints](/growth-ideas/structured-data-as-ai-citation-hints/) - same source, 2 shared channels, 2 shared stages
- [Experience-backed content moat](/growth-ideas/experience-backed-content-moat/) - same source, 2 shared channels
- [Well-known llms aliases for agent compatibility](/growth-ideas/well-known-llms-aliases-for-agent-compatibility/) - 2 shared channels, 2 shared stages
- [llms discovery headers on every page](/growth-ideas/llms-discovery-headers-on-every-page/) - 2 shared channels, 2 shared stages

## Read GrowthDex essays

Browse the plain-English essay index at [GrowthDex Blog](/blog/).

## Related GrowthDex essays

- [The AI discovery surface should teach the crawler and the agent](/blog/the-ai-discovery-surface-should-teach-the-crawler-and-the-agent/) - ai discovery, technical seo, developer marketing

## Advisory

If you want help turning this into a working growth system, Ian Goh offers advisory at https://iangoh.com/advisory.