Pillar 1 — Crawl & Discovery
Layer A — Agent Readiness. Owner: Engineering. Weight: 20 pts.
Items in this pillar
Section titled “Items in this pillar”| # | Item | Status | Authority |
|---|---|---|---|
| 01 | robots.txt with AI bot allowlist + Content Signals | required | RFC 9309 + Cloudflare convention; coordinating with IETF AI Preferences WG |
| 02 | sitemap.xml with valid <lastmod>, no thin pages | required | sitemaps.org |
| 03 | Canonical URLs on every indexable page | required | Cloudflare Redirects for AI Training (April 2026 beta) — incorrect canonicals get 301’d for verified training crawlers |
| 04 | Link response header (markdown alternate, canonical, next/prev) | recommended | RFC 8288 |
Full item descriptions: Universal Requirements § Pillar 1.
Where this gets enforced
Section titled “Where this gets enforced”Gate A — Technical checks A.1 (robots.txt + allowlist), A.2 (sitemap), and A.7 (Link header) against these items before any property ships.