What is Crawl Cost?

Definition

Crawl Cost (0–100, where lower is better) estimates the resource burden your pages impose on AI crawlers. Heavy pages consume more bandwidth, processing time, and token budget — making crawlers less likely to crawl your site thoroughly or frequently.

How SEODiff computes it

Crawl Cost is built from three additive penalty components:

• HTML size tier:
  • <50KB → 10 (lightweight)
  • <200KB → 30 (moderate)
  • <500KB → 50 (heavy)
  • ≥500KB → 70 (very heavy)
• Script penalty: >100KB inline scripts → +15; >50KB → +10
• Bloat penalty: Token bloat >30× → +15; >15× → +10

The final score is capped at 100.

Labels

• Low (score < 40): Lightweight, crawler-friendly pages.
• Medium (40–69): Some optimization possible.
• High (≥ 70): Heavy pages that crawlers may deprioritize.

Why it matters

AI crawlers have bandwidth budgets. If your pages are heavy, they'll crawl fewer of them — especially on programmatic sites with thousands of URLs. Lighter pages mean more of your content gets into AI training data and retrieval indexes.