Explicit AI-bot allowlist in robots.txt
Name the major AI crawlers in `robots.txt` and explicitly allow them instead of relying on a generic wildcard and hoping the agent interprets it the way you intended.
Why this can grow a startup
A lot of teams think access policy is obvious because the site is public. Agents do not infer policy from vibes. They look at `robots.txt`. When the file clearly allows the major AI crawlers, you remove a quiet discovery failure and make the site's stance easier for both machines and humans to verify.
Company example
Vercel's agent-readability spec recommends not blocking GPTBot, ClaudeBot, CCBot, or Google-Extended and says it is better to explicitly allow access than to have no robots policy at all.
Source and metric
Source: Vercel Knowledge Base · Browse Vercel Knowledge Base tactics
Vercel explicitly calls out GPTBot, ClaudeBot, CCBot, and Google-Extended as bots that should not be blocked
Source discovered: May 27, 2026
When to use it
Use this when AI Search, SEO, Website is relevant to ai discovery, technical seo, crawlability and you can run a bounded test with a free budget.
When not to use it
Do not use it as a substitute for customer evidence, a clear owner, or a measurable stop condition. Local platform rules and market behavior still need checking.
Founder checklist
- Read Vercel Knowledge Base and identify what is directly supported.
- Choose one channel context: AI Search, SEO, Website.
- Define the test around Vercel explicitly calls out GPTBot, ClaudeBot, CCBot, and Google-Extended as bots that should not be blocked.
- Set an owner, evidence window, and stop condition before launch.
Explore the context
Apply this with an operator
Turn isolated search tactics into a crawlable visibility system tied to demand and proof.