🤖 SEODiffBot/1.0

An honest, transparent AI readability crawler.

SEODiffBot measures how well websites serve content to AI systems — LLMs, search engines, and automated agents. It's part of the SEODiff AI Visibility Intelligence platform.

Quick Summary

User-Agent	`Mozilla/5.0 (compatible; SEODiffBot/1.0; +https://seodiff.io/bot)`
robots.txt name	`SEODiffBot`
From header	`[email protected]`
Crawl rate	≤ 1 request per second per domain
Respects robots.txt	Yes — always
Respects Retry-After	Yes — always
Purpose	AI readability measurement (ACRI scoring)
Operator	SEODiff
Contact	[email protected]
Machine-readable info	/bot.json

What does SEODiffBot do?

SEODiffBot visits the homepage of websites to measure their AI Content Readiness Index (ACRI) — a score from 0 to 100 that reflects how well a site's content can be understood by AI systems like ChatGPT, Claude, Perplexity, and Google's AI Overviews.

The crawler checks:

Whether content is available in raw HTML (vs. requiring JavaScript rendering)
Structured data (Schema.org, JSON-LD) quality and completeness
Semantic HTML usage (headings, lists, tables)
robots.txt policies for AI bots (GPTBot, ClaudeBot, etc.)
Technical signals: load time, HTTPS, meta tags, Open Graph

Crawl Behavior

Per-domain limits

Maximum 3 requests per domain: robots.txt, homepage HTML, and optionally a HEAD pre-flight check.
1 req/s rate limit per domain: We never flood a single site.
Global CDN-aware throttling: We detect Cloudflare, Fastly, Akamai, and other CDNs and limit concurrent requests per CDN network to avoid triggering WAF protections.

Circuit breaker

If we receive HTTP 429 responses, the crawler immediately:

Reads the Retry-After header and respects it
Pauses all requests to that CDN for at least 5 minutes
Uses exponential backoff on repeated rate limits (up to 30 min)
Does not penalize the affected domain's ACRI score

HEAD pre-flight

Before re-crawling a domain, we send a HEAD request to check Last-Modified. If the content hasn't changed since our last visit, we skip the full crawl entirely — saving both our bandwidth and yours.

How to control SEODiffBot

Block via robots.txt

User-agent: SEODiffBot
Disallow: /

We check robots.txt on every crawl and fully respect Disallow rules. If you block SEODiffBot, your site will not appear in our radar or receive an ACRI score.

Rate limit

Return a 429 Too Many Requests response with a Retry-After header and we will back off immediately.

Contact us

If you have concerns about our crawler, please email [email protected] and we'll address them promptly.

IP Addresses

SEODiffBot currently operates from a single dedicated server:

IPv4	`192.99.101.226`
IPv6 subnet	`2607:5300:60:62e2::/64` Rotating — 256 addresses
ASN	AS16276 (OVH SAS)
Location	Beauharnois, QC, Canada

Requests originate from the IPv4 above or any address in the 2607:5300:60:62e2::/64 range. If you see an IPv6 address starting with 2607:5300:60:62e2:, that is us.

The machine-readable version at /bot.json always has the latest IP allocations. We plan to apply for Cloudflare Verified Bot status in the future.

HTTP Headers

Every request from SEODiffBot includes these headers:

User-Agent: Mozilla/5.0 (compatible; SEODiffBot/1.0; +https://seodiff.io/bot)
From: [email protected]
X-Crawler-Purpose: AI Content Readability Index (ACRI) Audit
X-Crawler-Info: https://seodiff.io/bot
Accept: text/html,application/xhtml+xml,*/*

The From header follows RFC 7231 §5.5.1. Custom X-Crawler-* headers make us identifiable in raw access logs even when dashboard UIs truncate the User-Agent.

Data Usage

Data collected by SEODiffBot is used to:

Compute public ACRI scores visible on the SEODiff Radar
Power industry benchmarks and technology comparisons
Help site owners understand their AI visibility posture

We do not store or redistribute your full page content. We extract only metadata and structural signals needed for scoring.

Open Standards

SEODiffBot supports and checks for:

robots.txt — standard crawler access control
TDM-Reservation — EU Text and Data Mining opt-out header
ai.txt / llms.txt — emerging AI policy standards
Schema.org JSON-LD — structured data for AI understanding

Last updated: February 2026 · Back to SEODiff