# The AI discovery surface should teach the crawler and the agent > Why schema, crawl files, experience-backed writing, skill frontmatter, and one-command installs usually beat publishing one more invisible docs section. - Canonical HTML: https://growth.iangoh.com/blog/the-ai-discovery-surface-should-teach-the-crawler-and-the-agent/ - Published: 2026-05-28 - Updated: 2026-05-29T00:00:00Z - Categories: ai discovery, technical seo, developer marketing - Niches: AI products, developer tools, SaaS, API platforms, creator tools ## On this page - Give the machine a plain description of the page before it guesses from layout - Make the crawl map obvious instead of assuming the bot will discover it the hard way - A page becomes harder to replace when it carries operator reality - Agents need operating constraints, not just prose - Discovery should collapse into action while intent is still warm ## Start with these related tactics - [Structured data as AI citation hints](/growth-ideas/structured-data-as-ai-citation-hints/): Mark up core pages with schema so search engines and answer systems can tell whether they are looking at a site, article, dataset, or author page before they guess from layout. - [Sitemap plus robots discovery pack](/growth-ideas/sitemap-plus-robots-discovery-pack/): Ship `/sitemap.xml` and `/robots.txt` together so crawlers can find the important routes fast instead of discovering the site only through navigation and luck. - [Experience-backed content moat](/growth-ideas/experience-backed-content-moat/): Attach first-hand operator evidence, examples, and constraints to every guide so the page reads like lived work rather than a polished summary of what everybody already knows. Most AI-discovery work breaks because the team publishes a file, checks a box, and assumes the machine now understands the product. Usually it does not. The crawler still needs a clean map. The answer engine still needs explicit clues about what the page is, who wrote it, and why it deserves a citation. The agent still needs instructions concrete enough to use the product without improvising itself into a wall. That is why the useful surface is not one artifact. It is a small pack of pages and instructions that make the same product legible from different angles. ## Give the machine a plain description of the page before it guesses from layout The first move is [structured data as AI citation hints](/growth-ideas/structured-data-as-ai-citation-hints/). I like this because it asks a dull but important question: if a crawler strips away the design, can it still tell whether this page is a guide, a dataset, a company profile, or a random marketing slab? I would pair that with [source-dated technique dataset](/growth-ideas/source-dated-technique-dataset/). One tells the machine what the page is. The other gives it fields sturdy enough to quote, sort, and trust later. ## Make the crawl map obvious instead of assuming the bot will discover it the hard way [sitemap plus robots discovery pack](/growth-ideas/sitemap-plus-robots-discovery-pack/) looks basic until you inherit a site where half the value lives behind JavaScript, old paths, or inconsistent canonicals. A sitemap and robots file do not guarantee ranking. They do give crawlers a stable front door and a short list of what matters. That belongs near [answer-first source citation pages](/growth-ideas/answer-first-source-citation-pages/). One helps the system find the page. The other helps it understand why that page should be the answer once found. ## A page becomes harder to replace when it carries operator reality [experience-backed content moat](/growth-ideas/experience-backed-content-moat/) is the part most teams want to skip because it is slower. You have to bring examples, caveats, and a point of view. But that is exactly what makes the page harder to flatten into commodity sludge. If the article sounds like anybody could have written it, an answer engine has no reason to keep returning to you. I would read it next to [self-serve code audit for skeptical buyers](/growth-ideas/self-serve-code-audit-for-skeptical-buyers/). Both tactics reward pages that show their work instead of wrapping the claim in posture. ## Agents need operating constraints, not just prose [skill frontmatter with compatibility and tool constraints](/growth-ideas/skill-frontmatter-with-compatibility-and-tool-constraints/) matters because an agent can read a polished docs page and still fail immediately if the runtime, tools, or assumptions are hidden. Good frontmatter works like a pre-flight checklist. It tells the agent what environment it is in and what it should not invent. I would keep that close to [agent-skills manifest with sha256 integrity](/growth-ideas/agent-skills-manifest-with-sha256-integrity/). One makes the instructions legible. The other makes the fetched instructions easier to trust. ## Discovery should collapse into action while intent is still warm The closing move is [one-command skill install from docs URL](/growth-ideas/one-command-skill-install-from-docs-url/). This is one of those tiny product decisions that quietly changes adoption. If the reader has to reverse-engineer how to load your capability into their workflow, most of them will stop at admiration. If the docs end with one command, the surface stops being a brochure and starts being a tool. That sits beside [JWT login redirect for personalized API docs](/growth-ideas/jwt-login-redirect-for-personalized-api-docs/). Both remove the dead space between understanding and first use. This cluster fits AI products, developer tools, SaaS, API platforms, and creator tools that want to be discovered by search systems and used inside agent workflows. If I were tightening one this week, I would ask whether the machine can classify the page, whether it can find the important routes quickly, whether the writing contains real operator evidence, whether the instructions expose the actual constraints, and whether the next step to use the product is embarrassingly easy. If you want help turning your docs, crawl surface, and AI-discovery layer into something buyers and agents can actually use, the advisory CTA is here: [work with Ian Goh](https://iangoh.com/advisory). ## Related GrowthDex tactics - [Structured data as AI citation hints](/growth-ideas/structured-data-as-ai-citation-hints/) - SEO, AI Search - [Sitemap plus robots discovery pack](/growth-ideas/sitemap-plus-robots-discovery-pack/) - SEO, AI Search - [Experience-backed content moat](/growth-ideas/experience-backed-content-moat/) - SEO, Content, AI Search - [Skill frontmatter with compatibility and tool constraints](/growth-ideas/skill-frontmatter-with-compatibility-and-tool-constraints/) - AI Search, Documentation, Developer Experience - [One-command skill install from docs URL](/growth-ideas/one-command-skill-install-from-docs-url/) - AI Search, Documentation, Activation ## Essay chronology - [Newer essay: The changelog should prove the product keeps moving](/blog/the-changelog-should-prove-the-product-keeps-moving/) - release communication, brand trust, technical seo - [Older essay: Directories start working when the page fits the search](/blog/directories-start-working-when-the-page-fits-the-search/) - SEO, community-led growth, demand capture ## Keep reading - [The product should keep a visible pulse](/blog/the-product-should-keep-a-visible-pulse/) - developer marketing, launches, brand trust - [The docs route should feel boring even when the product is moving](/blog/the-docs-route-should-feel-boring-even-when-the-product-is-moving/) - docs strategy, technical seo, support-led growth - [The docs site should answer like one product](/blog/the-docs-site-should-answer-like-one-product/) - technical seo, docs strategy, brand trust ## Continue through the blog - [SaaS](/blog/#path-saas) - 3 essays in this path - [AI products](/blog/#path-ai-products) - 3 essays in this path - [developer tools](/blog/#path-developer-tools) - 3 essays in this path ## Sources - [Google Search Central: Introduction to structured data markup in Search](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data) · [GrowthDex source hub](/sources/google-search-central-introduction-to-structured-data-markup-in-search-d/) - [Google Search Central: Sitemaps overview](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) · [GrowthDex source hub](/sources/google-search-central-sitemaps-overview-developers-google-com/) - [Google Search Central: Creating helpful, reliable, people-first content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) · [GrowthDex source hub](/sources/google-search-central-creating-helpful-reliable-people-first-content-dev/) - [Mintlify Docs: skill.md](https://www.mintlify.com/docs/ai/skillmd) · [GrowthDex source hub](/sources/mintlify-docs-skill-md-mintlify-com/) ## Editing notes - Kept the essay on one practical claim: AI discovery is a pack of surfaces, not a single metadata trick. - Used ordinary objects like schema, sitemaps, robots files, frontmatter, and install commands instead of inflated AI-search language. - Linked each section to adjacent tactic pages so the reader can move from the essay into specific implementation routes. - Closed with an operator checklist and advisory CTA instead of a generic future-of-AI wrap-up. ## Advisory If you want help turning this into a growth system, Ian Goh offers advisory at https://iangoh.com/advisory.