AI SEO Crawlability & Keyword Dataset

Read full text Summarise with VIRAT
Authors: Sonju Ghosh
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

The dataset is designed to evaluate website crawlability and accessibility for AI bots and search engine crawlers. Each record represents a domain along with its crawlability score and permissions for various AI-driven bots. Dataset Fields ExplanationDomain The website domain being analyzed for crawlability and bot access.Crawlability Score (%) A percentage-based score indicating how accessible the website is for crawlers and AI bots. Higher scores represent better accessibility and SEO readiness.AI Insight AI-generated analysis highlighting crawl issues, restrictions, or optimization opportunities for the domain.Robots.txt Link Direct link to the website’s robots.txt file used to control crawler access. AI Bot Accessibility StatusThese fields indicate whether specific AI bots are Allowed / Blocked / Restricted based on robots.txt rules:GPTBot Status Access status for OpenAI’s web crawler.ClaudeBot Status Access status for Anthropic’s Claude AI crawler.PerplexityBot Status Indicates whether Perplexity AI can crawl the site.Google-Extended Status Controls access for Google’s AI training crawler.CCBot Status Access for Common Crawl bot (used in many AI datasets).anthropic-ai Status Additional Anthropic-related crawler access.OAI-SearchBot Status Status of OpenAI’s search-focused crawler.⚙️ Purpose of the DatasetThis dataset helps to:Analyze AI bot crawl permissions across websitesIdentify restrictions in robots.txt affecting AI visibilityMeasure overall crawlability score for SEO and AI indexingOptimize websites for AI search engines and LLM visibilityDetect opportunities to improve AI-driven traffic and discoverability Use CasesAI SEO optimizationLLM (ChatGPT, Claude, Perplexity) visibility analysisrobots.txt audit and fixingCompetitive analysis of AI crawl accessBuilding AI-ready websites Website: https://dofollo.ai/

Fields of Study & Subjects

Web visibility and informetrics Web Data Mining and Analysis Information Retrieval and Search Behavior

Publication details

Journal / source
Figshare
Publication date
28 March 2026
Language
EN
Document type
Dataset