AI SEO Crawlability & Keyword Dataset
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
The dataset is designed to evaluate website crawlability and accessibility for AI bots and search engine crawlers. Each record represents a domain along with its crawlability score and permissions for various AI-driven bots. Dataset Fields ExplanationDomain The website domain being analyzed for crawlability and bot access.Crawlability Score (%) A percentage-based score indicating how accessible the website is for crawlers and AI bots. Higher scores represent better accessibility and SEO readiness.AI Insight AI-generated analysis highlighting crawl issues, restrictions, or optimization opportunities for the domain.Robots.txt Link Direct link to the website’s robots.txt file used to control crawler access. AI Bot Accessibility StatusThese fields indicate whether specific AI bots are Allowed / Blocked / Restricted based on robots.txt rules:GPTBot Status Access status for OpenAI’s web crawler.ClaudeBot Status Access status for Anthropic’s Claude AI crawler.PerplexityBot Status Indicates whether Perplexity AI can crawl the site.Google-Extended Status Controls access for Google’s AI training crawler.CCBot Status Access for Common Crawl bot (used in many AI datasets).anthropic-ai Status Additional Anthropic-related crawler access.OAI-SearchBot Status Status of OpenAI’s search-focused crawler.⚙️ Purpose of the DatasetThis dataset helps to:Analyze AI bot crawl permissions across websitesIdentify restrictions in robots.txt affecting AI visibilityMeasure overall crawlability score for SEO and AI indexingOptimize websites for AI search engines and LLM visibilityDetect opportunities to improve AI-driven traffic and discoverability Use CasesAI SEO optimizationLLM (ChatGPT, Claude, Perplexity) visibility analysisrobots.txt audit and fixingCompetitive analysis of AI crawl accessBuilding AI-ready websites Website: https://dofollo.ai/
Fields of Study & Subjects
Web visibility and informetrics
Web Data Mining and Analysis
Information Retrieval and Search Behavior
Publication details
- Journal / source
- Figshare
- Publication date
- 28 March 2026
- Language
- EN
- Document type
- Dataset