Tool: Crawler Health

What it does

Crawler Health simulates access from 5 AI bot user agents and produces a per-bot status report plus an overall health score.

Bots tested

Bot	User Agent String	Operator
GPTBot	GPTBot/1.0	OpenAI
ClaudeBot	ClaudeBot/1.0	Anthropic
PerplexityBot	PerplexityBot/1.0	Perplexity
GoogleBot	Googlebot/2.1	Google
Applebot	Applebot/0.1	Apple

How it works

Step 1: Robots.txt check

Fetches and parses /robots.txt to check if each bot's user agent is disallowed. This uses standard robots.txt parsing rules.

Step 2: HTTP fetch with UA spoofing (Pro)

On Pro plans, the tool makes an actual HTTP request using each bot's user agent string. This detects WAF/CAPTCHA blocks that only appear for specific user agents but allow browsers.

Step 3: JavaScript rendering heuristic

Checks if the page uses heavy JavaScript frameworks that would require a rendering engine that most AI bots don't have.

Scoring

Component	Score Impact
Base score	100
Per blocked bot	−15
WAF/CAPTCHA detected	−10
Heavy JS (client rendering)	−20

This feeds into the Bot Access Score of the AI Readiness radar.

JSON output fields

• overall_score — 0–100 health score
• bots — array of per-bot results with name, allowed, status_code, blocked_reason
• waf_detected — boolean, true if WAF/CAPTCHA was detected
• js_heavy — boolean, true if heavy JS framework detected

API endpoint

GET /api/crawler-health?url=https://example.com