WebSweep: a Python library for high-throughput web scraping for researchers

WebSweep is a Python library for high-throughput web scraping for researchers.It is designed to stay simple for beginners while still handling large URL lists.The primary objective is to run effectively on a single computer (laptop or workstation),without requiring cloud infrastructure or distributed orchestration. It is built for projects that start with a list of websites and need a workflowthat is easy to rerun, inspect, and extend: - crawl websites from a list of base URLs- follow only within-domain links up to a bounded depth- extract page-level text and metadata- consolidate results back to one record per domain- (if desired) repeat the same sweep monthly or quarterly with the same configured instance The goal is research infrastructure, not cloud orchestration. WebSweep is meantto work well on a laptop or workstation, with intermediate outputs that are easyto archive, validate, and analyse later. ## What WebSweep Is Good For WebSweep fits best when you want to study many websites in a comparable,repeatable way. Typical research uses: - track how organisations discuss a topic over time across many domains- build corpora from university, company, NGO, or government websites- monitor recurring updates on the same website lists every few months- extract page text plus a few structured fields for downstream analysis It is especially useful when the unit of analysis is the domain or organisation,not one single large site. WebSweep is probably **not** the right tool when: - you need to interact with JavaScript-heavy websites- you want to scrape one very complex website with highly custom logic- you need browser automation rather than HTML crawling For those cases, tools such as [Scrapy](https://github.com/scrapy/scrapy) or[Selenium](https://pypi.org/project/selenium/) may be a better fit. ## Install ```bashpip install websweep``` ## What You Need WebSweep needs a list of URLs. - CLI mode: CSV or TSV file with header (`url`, optional `identifier`)- Library mode: Python list of URLs or `(url, identifier)` tuples Example CSV: ```csvurl,identifierhttps://example.com,examplehttps://example2.org,example_org``` ## Choose Your Mode - **CLI mode** (`websweep ...`): easiest way to run repeatable instance-based crawls from a CSV/TSV source file.- **Library mode** (`from websweep import ...`): best when you need custom Python logic (custom extractors, analysis loops, notebooks). ## Workflow  ```textInput URLs -> Crawler In: URL list + crawl settings Out: crawled_data/*.zip (zipped pages per domain) overview_urls.{duckdb|db|tsv} (per-page crawl status overview) -> Extractor In: overview file + crawled_data/*.zip Out: extracted_data/*.ndjson (extracted data per web page) -> Consolidator In: extracted_data/*.ndjson Out: consolidated_data/*.ndjson (consolidated to domain level)``` One-pass mode (lower disk usage): ```textInput URLs -> Crawler(extract=True, save_html=False) -> extracted_data/*.ndjson``` ### What Each Step Does - `Crawler`: starts from base URLs (one domain per row), downloads pages, follows only within-domain links, applies exclusion rules (for example blocked extensions), and stops at depth `max_level` (default: `3`).- `Extractor`: reads crawled pages and extracts structured page-level fields such as cleaned text (`text`), metadata (`meta_*`), and location fields (`zipcode`, `address`).- `Consolidator`: merges page-level records back to one record per domain, keeping concatenated domain text and aggregated information (e.g., `zipcode` frequencies, where the most frequent can be treated as the main postcode and others as additional postcodes) ## Quickstart (Python) ```pythonfrom pathlib import Pathfrom websweep import Crawler, Extractor, Consolidator urls = [ "https://www.dggrootverbruik.nl/", "https://www.gosliga.nl/", "https://www.heeren2.nl/",] out = Path("./research_output") # 1) CrawlCrawler(target_folder_path=out).crawl_base_urls(urls) # 2) ExtractExtractor(target_folder_path=out).extract_urls() # 3) ConsolidateConsolidator(target_folder_path=out).consolidate()``` ## Quickstart (CLI) ```bashwebsweep init --headlesswebsweep crawlwebsweep extractwebsweep consolidate```