CRAWLDoc: A System for Contextual Ranking and Bibliographic Metadata Extraction from Web Resources

Publication databases rely on accurate metadata extraction from diverse web sources, yet variations in web layouts and data formats present challenges for metadata providers. This paper introduces CRAWLDoc, a novel system for contextual ranking of linked documents and metadata extraction from web resources. Using a publication’s DOI, CRAWLDoc retrieves the landing page and associated linked documents, including PDFs, ORCID profiles, and supplementary materials. It embeds these documents, along with anchor texts and URLs, into a unified representation. Our layout-independent embedding and ranking system ensures robustness across various web layouts and formats. Experimental results demonstrate CRAWLDoc’s superior performance in extracting bibliographic metadata compared to relying solely on landing pages. A leave-one-out experiment across six publishers shows CRAWLDoc’s resilience to layout changes and consistent extraction accuracy. Our source code and dataset can be accessed at https://github.com/FKarl/crawldoc-metadata-extraction