Blocked by robots.txt

Common robots.txt Mistakes

• Blocking entire directories. Disallow: /blog/ when you only meant to block one page.
• Wildcard overkill. Disallow: /*? blocks all URLs with query parameters, including legitimate paginated content.
• Staging rules in production. Disallow: / left over from a dev/staging environment. This blocks the entire site.
• Blocking CSS/JS resources. Google needs access to CSS and JS files to render pages. Blocking /wp-content/ or /static/ breaks rendering.
• User-agent mismatches. Rules targeting User-agent: Googlebot vs User-agent: * behave differently. Typos in user-agent names silently fail.

How to Fix It

1. Audit your robots.txt. Use GSC's robots.txt tester to verify which URLs are blocked.
2. Use SEODiff's Crawl Access Checker to test access for Googlebot, GPTBot, ClaudeBot, and other crawlers simultaneously.
3. Be specific. Instead of blocking entire directories, use precise path patterns.
4. Allow CSS/JS/images. Never block /static/, /assets/, or /wp-content/themes/.
5. Test before deploying. Always test robots.txt changes against your URL list before pushing to production.

robots.txt vs meta robots vs noindex

These are different mechanisms with different effects:

• robots.txt Disallow — Prevents crawling. Google can't see the page at all. If external links point to it, Google may still index the URL (with no snippet).
• meta robots noindex — Allows crawling but prevents indexing. Google sees the page but won't show it in results.
• X-Robots-Tag: noindex — Same as meta robots, but set via HTTP header. Works for non-HTML files (PDFs, images).

Important: If you block a page in robots.txt AND have a noindex meta tag, Google will never see the noindex tag (because it can't crawl the page). The noindex is effectively invisible.