Explicit AI-bot allowlist in robots.txt
Name the major AI crawlers in `robots.txt` and explicitly allow them instead of relying on a generic wildcard and hoping the agent interprets it the way you intended.
Why this can grow a startup
A lot of teams think access policy is obvious because the site is public. Agents do not infer policy from vibes. They look at `robots.txt`. When the file clearly allows the major AI crawlers, you remove a quiet discovery failure and make the site's stance easier for both machines and humans to verify.
Source-backed example
Vercel's agent-readability spec recommends not blocking GPTBot, ClaudeBot, CCBot, or Google-Extended and says it is better to explicitly allow access than to have no robots policy at all.
Source-reported evidence
Source: Vercel Knowledge Base · Browse Vercel Knowledge Base tactics
Vercel explicitly calls out GPTBot, ClaudeBot, CCBot, and Google-Extended as bots that should not be blocked
Attributed to the cited source; GrowthDex has not independently verified it. Use this as context, not as a forecast, benchmark, or recommended target.
Evidence scope: claims and metrics are attributed to the linked source; GrowthDex has not independently replicated the reported result. Check the original context and validate fit before applying it. Read the evidence methodology.
Source discovered: May 27, 2026
When to use it
Use this when AI Search, SEO, Website is relevant to ai discovery, technical seo, crawlability and you can run a bounded test with a free budget.
When not to use it
Do not use it as a substitute for customer evidence, a clear owner, or a measurable stop condition. Local platform rules and market behavior still need checking.
How to do it
- Choose one target segment and define the exact next action this tactic should prompt.
- Adapt the tactic to AI Search and SEO, using evidence from Vercel Knowledge Base as context—not as a promised result.
- Record a baseline for: Qualified search visitors who complete the intended next action. Set a guardrail that would make you pause the test.
- Run a narrow pilot and choose a test window that fits the normal decision or use cycle; do not scale before you have observed the full cycle.
- Compare the signal with its baseline. Scale only if the intended action improves without crossing the guardrail; otherwise revise one variable or stop and record what you learned.
Compare a stack for your situation
Compare source-backed tactics against your product, stage, budget and channel strengths, then choose one experiment to test.
Explore the context
Did you launch this experiment?
Report a launch to help us understand which growth ideas founders put into practice.
Apply this with an operator
Turn isolated search tactics into a crawlable visibility system tied to demand and proof.
SOURCE HUB
More tactics from Vercel Knowledge Base
Explore related GrowthDex examples and evidence connected to Vercel Knowledge Base.
Open Vercel Knowledge Base source hub