# Explicit AI-bot allowlist in robots.txt

> Name the major AI crawlers in `robots.txt` and explicitly allow them instead of relying on a generic wildcard and hoping the agent interprets it the way you intended.

- Canonical HTML: https://growth.iangoh.com/growth-ideas/explicit-ai-bot-allowlist-in-robots-txt/
- Source: [vercel.com](https://vercel.com/kb/guide/agent-readability-spec)
- GrowthDex source hub: [Vercel Knowledge Base](/sources/vercel-knowledge-base-vercel-com/)
- Last checked: 2026-05-27
- Rarity: uncommon
- Budget: free
- Channels: AI Search, SEO, Website
- Stages: ai discovery, technical seo, crawlability, brand trust


## Why this can grow

A lot of teams think access policy is obvious because the site is public. Agents do not infer policy from vibes. They look at `robots.txt`. When the file clearly allows the major AI crawlers, you remove a quiet discovery failure and make the site's stance easier for both machines and humans to verify.

## Ian's take

From scaling consumer platforms across MENA and Southeast Asia, my default is to distrust growth work that only looks good in a slide. For SEO and AI search, I care less about clever keyword tricks and more about clarity. A buyer, crawler, or answer engine should quickly understand who this is for, why it works, what proof backs it, and what page deserves to be cited. I would run it small enough to learn quickly, then only scale the parts that real users repeat, save, reply to, or buy from. For this tactic, I would watch one clear growth signal before putting more time or budget behind it.

## Action plan

1. Define one narrow startup segment where explicit ai-bot allowlist in robots.txt can create a measurable lift.
2. Turn the tactic into one offer, page, campaign, or workflow for the AI Search and SEO channel.
3. Use the evidence from vercel.com to set the first version of the message, format, and audience.
4. Launch a small test for 7 to 14 days with one success metric: one measurable growth signal.
5. Review the result, keep the winning message, remove weak variants, and turn the learning into a repeatable growth playbook.

## Source-backed example

Vercel's agent-readability spec recommends not blocking GPTBot, ClaudeBot, CCBot, or Google-Extended and says it is better to explicitly allow access than to have no robots policy at all.

## Adjacent tactics in the same lane

- [sitemap.md semantic discovery map](/growth-ideas/sitemap-md-semantic-discovery-map/) - same source, 3 shared channels
- [Content-negotiated markdown on canonical URLs](/growth-ideas/content-negotiated-markdown-on-canonical-urls/) - same source, 3 shared channels
- [Well-known llms aliases for agent compatibility](/growth-ideas/well-known-llms-aliases-for-agent-compatibility/) - 3 shared channels, 3 shared stages
- [llms discovery headers on every page](/growth-ideas/llms-discovery-headers-on-every-page/) - 3 shared channels, 3 shared stages

## Read GrowthDex essays

Browse the plain-English essay index at [GrowthDex Blog](/blog/).

## Related GrowthDex essays

- [An agent needs a capability page, not just a crawl map](/blog/an-agent-needs-a-capability-page-not-just-a-crawl-map/) - AI Search, brand trust, technical SEO

## Advisory

If you want help turning this into a working growth system, Ian Goh offers advisory at https://iangoh.com/advisory.