Maven Dependency Crawler (Anonymous Version)

Maven-Dependency-Crawler — Maven Dependency Metadata Dataset This repository provides a reproducible pipeline for constructing a unified dataset of Maven artifacts and their dependency metadata. The crawlers query multiple major public artifact repositories and aggregate the results into a single structured JSON dataset suitable for software dependency research, machine learning models, and large-scale ecosystem analysis. The final dataset generated by this project is MavCrawl_dataset.json, located at the repository root. Data Sources This dataset integrates artifact metadata from: Atlassian public Maven repository Cloudera Maven repository Google-hosted Gradle/Maven artifacts Maven Central Each source provides complementary or overlapping information. The pipeline normalizes, merges, and deduplicates entries across sources. Dataset Contents Each record in MavCrawl_dataset.json corresponds to one artifact and may include: _id - Unique identifier (group:artifact:version) origin_repository - One or more repositories reporting the artifact last_modified - Last known artifact update timestamp (string or "Unknown") jar_size - Reported binary size (string or "Unknown") description - Artifact summary or "Unknown" direct_dependencies - Resolved dependencies in the form group:artifact:version:scope source_code_url - Repository or VCS location if available parent_module - Parent artifact for multi-module builds (if applicable) child_modules - Child modules defined by the project (if applicable) origin_repository - Repository from which artifact data is sourced extraction_status - Artifact extraction status Values may be "Unknown" when metadata is unavailable from the source repository. Reproducibility Workflow Ensure Python dependencies are installed (requirements.txt). Create a .env file containing a MONGO_URI connection string. Run the individual crawlers:python atlassian_repo_crawler/atlassianCrawler.pypython cloudera_repo_crawler/cloudEraCrawler.pypython google_repo_crawler/google_crawler.pypython mavenCentral_repo_crawler/mavenCrawler.py Merge results into the final dataset: python combine_datasets.py Purpose and Research Context This dataset is intended for work involving: Dependency conflict detection and resolution Ecosystem evolution and artifact lineage studies Learning-based or recommender-style dependency version selection Empirical analysis of Java build dependency graphs It has been used in downstream machine learning research on automated dependency version recommendation. Full Documentation and Source Code Complete instructions, crawler internals, and dataset schema details are available in the GitHub repository: https://anonymous.4open.science/r/Maven-Dependency-Crawler-98A6