How it's calculated
Crawl Cost is an additive score from three penalty components (0–100, where lower = better):
- HTML size tier: <50KB → 10, <200KB → 30, <500KB → 50, ≥500KB → 70
- Script penalty: >100KB inline scripts → +15, >50KB → +10
- Bloat penalty: Token bloat >30× → +15, >15× → +10
Labels
- Low (score < 40): Lightweight, crawler-friendly pages.
- Medium (40–69): Room for optimisation but not critical.
- High (≥ 70): Heavy pages that crawlers may deprioritise or skip.
Why it matters
AI crawlers have bandwidth and time budgets. Heavy pages reduce crawl coverage — especially on large programmatic sites with thousands of URLs. Keeping pages lightweight means more of your content gets into AI training and retrieval indexes.