The Science of ACRI

Executive Summary

Conservative interpretation: these results show a strong association between ACRI and retrieval in our Shadow RAG pilot; further work is required to confirm transfer to commercial embedding stacks and multi-page indices.

• ACRI predicts retrieval. The AI-Crawler Reality Index is a statistically significant predictor of whether a page will be selected by a RAG (Retrieval-Augmented Generation) system.
• Bot Access (GPTBot) is the #1 retrieval factor. Our ablation study shows that this signal has the largest impact on retrieval probability (Δρ = +0.2291).
• Schema Coverage ranks second. The second most impactful signal contributes Δρ = +0.1407 to the ACRI–retrieval correlation.
• Content quality drives topical retrieval. When queries don't contain the domain name, only sites with rich, specific, well-structured content are retrieved — exactly the sites with high ACRI scores.
• Bot access is the gateway. Blocking AI crawlers (GPTBot, ClaudeBot) eliminates content from the retrieval pipeline entirely — it's a binary prerequisite. Once access is granted, the quality signals (Schema, Content Depth, Ghost Ratio) determine retrieval ranking.

1. Introduction & Motivation

The Crisis of Token Noise

The modern web was built for browsers, not for AI. JavaScript-heavy frameworks, CSS-in-JS, client-side rendering, and bloated HTML create a signal-to-noise disaster for LLM context windows. When an AI system like ChatGPT, Perplexity, or a custom RAG pipeline ingests a web page, it doesn't see what a human sees — it sees raw HTML, scripts, and noise.

This matters because Retrieval-Augmented Generation (RAG) systems now power a growing share of information retrieval. These systems:

1. Embed web content into vector representations
2. Store these vectors in a database (FAISS, Pinecone, etc.)
3. Retrieve the most relevant vectors for a user query
4. Generate an answer from the retrieved content

If your content is bloated, poorly structured, or inaccessible to AI crawlers, it will produce low-quality embeddings, rank poorly in retrieval, and be excluded from AI-generated answers. Google ranking ≠ AI visibility.

Why ACRI?

The AI-Crawler Reality Index (ACRI) was designed to measure how "AI-visible" a website truly is. Unlike traditional SEO scores that focus on human-facing signals, ACRI quantifies the four pillars that determine whether AI systems can extract, understand, and cite your content.

But until now, ACRI has been a theoretical framework. This study provides the empirical proof: does ACRI actually predict retrieval success?

2. The ACRI Framework

ACRI is a composite score (0–100) computed as a weighted geometric mean of four pillar sub-scores:

Weights derived from calibration studies on 100k+ domains (see Section 6).

The geometric mean penalizes weak areas: a site with excellent schema but terrible token bloat will score lower than one with balanced, moderate scores across all pillars. This reflects the reality that AI systems need all signals to work together.

Glossary of Proprietary Metrics

The "Golden Semantic String"

When an AI system ingests a page, it attempts to extract the core meaning. We call this the Golden Semantic String. It is the canonical text an AI system would extract (title, meta description, H1, H2s, first 300–600 words, JSON-LD).

<div class="flex-col w-full">
  <script>window.__INITIAL_STATE__={...}</script>
  <style>.btn-primary{color:blue;}</style>
  <!-- 400 lines of SVG icons -->
  <h1 class="text-2xl font-bold">Our Product</h1>
  <p>Buy now.</p>
</div>
    

# Our Product
Buy now.

[Schema: Product, Brand: SEODiff]
    

Grade Scale

3. Methodology: Shadow RAG Calibration

Reproducibility note: all code, sample data, and pipeline scripts are published in github.com/seodiff/research. This pilot uses all-MiniLM-L6-v2; robustness checks with stronger/open and commercial embeddings are tracked in the V3 roadmap.

3.1 What is a Shadow RAG?

A Shadow RAG is a controlled, private replica of the retrieval pipelines used by AI systems like Perplexity, ChatGPT (Browse), and SearchGPT. By building our own retrieval system and testing it against known ground-truth, we can measure exactly how well ACRI predicts real-world retrieval success — without relying on opaque third-party APIs.

3.2 Dataset & Sampling

3.3 Embedding Pipeline

1. Text extraction: For each domain, the Golden Semantic String is extracted (title | meta description | H1 | H2 headings | content body).
2. Embedding model: all-MiniLM-L6-v2 (sentence-transformers). Free, open-source, and widely used in RAG research. Produces 384-dimensional embeddings that mimic the behavior of commercial embedding APIs.
3. Normalization: All embeddings are L2-normalized to enable cosine similarity via inner product search.
4. Vector index: FAISS (Facebook AI Similarity Search) with flat inner-product index for exact nearest-neighbor search.

3.4 Query Battery

We generated 4630 queries (5 per domain, across 3 query types) that reflect real-world LLM usage patterns:

Each query has an explicit ground-truth mapping to the correct domain, enabling precise measurement of retrieval accuracy.

3.5 Evaluation Metrics

We compute Recall@1, Recall@5, and Recall@10, plus Mean Reciprocal Rank (MRR). Statistical significance is assessed via Spearman ρ with 1,000-iteration bootstrap confidence intervals.

3.6 Statistical Tests

• Spearman ρ: Rank correlation between ACRI score and Retrieval Score (robust to outliers)
• Pearson r: Linear correlation for comparison
• Partial correlation: Controlling for domain authority (log Tranco rank) to isolate the ACRI effect from popularity bias
• Bootstrap CI: 1,000 resamples for 95% confidence intervals on all correlations
• Ablation study: Remove each ACRI sub-signal and measure Δρ to estimate feature importance

4. Experimental Results

4.1 Global Metrics

4.2 Correlation: ACRI → Retrieval

Figure 1: Scatter plot of ACRI score vs. Recall@10 with regression line. Each dot represents one domain. Higher ACRI scores correlate with higher retrieval success.

4.3 Traditional Authority vs AI Visibility

Figure 2: Traditional authority proxy (log10 Tranco rank) vs Recall@10. Circled points are high-authority domains with low AI retrieval, supporting the claim that Google ranking ≠ AI visibility.

4.4 Metrics by Query Type

Figure 3: Mean Reciprocal Rank broken down by query type. Entity queries have the highest retrieval success, while comparison queries are more challenging.

5. Ablation Study: Feature Importance

To understand which ACRI sub-signals matter most for retrieval, we performed an ablation study. For each signal, we residualized ACRI (removed the linear contribution of that signal via linear regression) and recomputed the Spearman ρ with Recall@10. The difference (Δρ) indicates how much that signal contributes to the overall ACRI–retrieval correlation.

Figure 4: Feature importance from ablation study. Token Bloat and Schema Coverage have the largest impact on ACRI's ability to predict retrieval success.

6. ACRI Tier Analysis

We split domains into five ACRI tiers and compared their average retrieval success.

Figure 5: Average Recall@10 by ACRI tier. Domains in the peak tier (70-89 (B/A)) are retrieved 5.6× more often than those in the bottom tier (0–29). The slight dip in the 90-100 tier is due to a smaller sample size (n=126 vs n=200).

7. Practical Implications

For Engineers & SEOs

For Product Teams Using ACRI

Checklist: Improve Your ACRI in 30 Minutes

8. Limitations & Future Work

Future Work

• Scale to 10k–100k domain index for more statistical power
• Test with multiple embedding models (all-mpnet-base-v2, OpenAI text-embedding-3-small)
• Add multi-page indexing per domain
• Introduce paraphrase robustness testing
• Run longitudinal study over 30–90 days to measure temporal stability
• Add retrieval-augmented answer generation with faithfulness scoring (Judge LLM)

9. Reproducibility & Data

Corpus Size:     926 domains
Queries:         4630
Embedding Model: all-MiniLM-L6-v2
FAISS Index:     flat_ip (exact cosine similarity)
Recall@K:        K = 1, 5, 10
Bootstrap:       1,000 iterations
Random Seed:     42
Timestamp:       2026-02-19T21:18:21Z
  

How to Reproduce

# 1. Install dependencies
cd shadow_rag/
pip install -r requirements.txt

# 2. Export domain data (needs API access or local JSONL)
python data_export.py

# 3. Run the full pipeline
python run_pipeline.py

# 4. Or run individual steps
python embedding.py          # Embed + index
python query_generator.py    # Generate queries
python evaluation.py         # Run retrieval + stats
python charts.py             # Generate charts
python whitepaper.py         # Render this document

# 5. One-command reproducibility run
sh run_pipeline.sh

# 6. Open Figure 1 source data + chart
python -c "import pandas as pd; print(pd.read_csv('output/domain_retrieval_scores.csv').head())"

Public reproducibility repo: github.com/seodiff/research

Artifacts Produced

• output/sampled_corpus.csv — Stratified domain sample with ACRI scores
• output/shadow_rag.faiss — FAISS vector index
• output/query_battery.csv — Full query set with ground truth
• output/domain_retrieval_scores.csv — Per-domain Retrieval Score
• output/experiment_results.json — Complete results with all metrics
• output/logs/retrieval_results.csv — Raw retrieval logs for every query
• output/charts/*.png — Publication-quality charts

Library Versions

sentence-transformers  5.2.3
faiss-cpu              1.13.2
numpy                  2.4.1
scipy                  1.17.0
pandas                 2.2.3
scikit-learn           1.8.0

10. Appendix: Ground Truth & Residualization

10.1 Ground-Truth Mapping Rules

10.2 Ablation Δρ (Ap) Residualization

To compute Ap for each signal, we regress ACRI on that signal, take residuals, and recompute Spearman between residualized ACRI and RetrievalScore.

# toy pseudocode
rho_base = spearman(acri, retrieval)
acri_resid = residualize(acri, signal)
rho_without_signal = spearman(acri_resid, retrieval)
delta_rho = rho_base - rho_without_signal

10.3 Statistical Reporting Notes

• Headline statistics are reported with bootstrap 95% confidence intervals where available.
• For families of multiple comparisons (e.g., ablation/subgroup screens), results should be interpreted with FDR-adjusted p-values in follow-up analyses.
• Power is strongest at global-correlation level and weaker for small subgroup slices; subgroup estimates are presented as directional unless otherwise noted.

About the Authors

SEODiff is the industry standard for AI Visibility analytics. We help engineering and marketing teams measure, monitor, and improve how their content is retrieved by LLMs (ChatGPT, Perplexity, Claude, SearchGPT).

This research was conducted by the SEODiff Data Science team to validate the ACRI framework, which powers our core auditing platform. Unlike black-box SEO tools, we believe in radical transparency: our methodology, datasets, and calibration studies are open for peer review.

Where do you sit on the curve?

Every day your site is blocked or bloated, you are invisible to the next generation of search. Run a free, instant Shadow RAG simulation on your domain now.

Free analysis • No credit card required • Instant results

Continue Reading

View all papers →