Fern custom robots policy before crawler ambiguity
Write an explicit `robots.txt` policy before assuming search engines and AI crawlers will interpret a default wildcard the way you meant.
Why this can grow a startup
A lot of teams treat crawl policy like a footnote, then wonder why discovery looks inconsistent across bots. Fern's robots guide is useful because it makes the policy layer concrete: named crawler rules, AI usage signals, sitemap restoration, and a reminder that private content still needs authentication. That turns `robots.txt` from a forgotten file into a trust and distribution control point.
Company example
Fern lets docs teams point `agents.robots-txt` at a custom file, keep named bot directives ahead of wildcard groups, and restore the sitemap line when replacing the default robots file.
Source and metric
Source: Fern Docs: Custom robots.txt
Fern's custom file example includes named crawler rules, a `Content-Signal` line, and an explicit `Sitemap` entry, while Fern still appends a managed block for internal routes.
When to use it
Use this when SEO, AI visibility, Brand Trust is relevant to robots txt, crawler policy, ai bots and you can run a bounded test with a free budget.
When not to use it
Do not use it as a substitute for customer evidence, a clear owner, or a measurable stop condition. Local platform rules and market behavior still need checking.
Founder checklist
- Read Fern Docs: Custom robots.txt and identify what is directly supported.
- Choose one channel context: SEO, AI visibility, Brand Trust.
- Define the test around Fern's custom file example includes named crawler rules, a `Content-Signal` line, and an explicit `Sitemap` entry, while Fern still appends a managed block for internal routes..
- Set an owner, evidence window, and stop condition before launch.
Explore the context
Apply this with an operator
Turn isolated search tactics into a crawlable visibility system tied to demand and proof.