Reproducible Experiments with Large-Scale Data via Dataset Splitting and Reconstruction

Read full text Summarise with VIRAT
Authors: Anonymous, Anonymous
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

SciConv addresses a key challenge in computational reproducibility: scientific datasets frequently exceed the per-record size limits imposed by data repositories such as Zenodo (50 GB per record). Without automated tooling, researchers must manually split, upload, and document dataset chunks , a process that is error-prone and difficult for others to reconstruct. This tool implements an automated pipeline for dataset splitting and reconstruction. When a dataset exceeds the repository limit, it is first archived into a single tar file to preserve directory structure, then split into fixed-size chunks (≤ 50 GB each). Each chunk is published as a separate Zenodo record with consistent metadata and cross-referenced identifiers recorded in a reproducibility manifest. At reproduction time, the manifest is used to locate and download all chunks in order, reconstruct the original archive, and extract the dataset - restoring the exact directory structure expected by the experiment code. Chunk integrity is verified using MD5 checksums stored in the manifest. The reconstruction process is fully automated through the included run scripts, requiring only Docker and a network connection from the reproducing researcher. Requirements: Python 3.10+ Docker Desktop Zenodo personal access token with deposit:write and deposit:actions scopes Installation: # Backend cd backendChat pip install -r requirements.txt cp .env.example .env.local # Set ZENODO_API_TOKEN in .env.local python main.py # Frontend (separate terminal) cd chatGPTWeb npm install npm run startdev The tool is then accessible at http://localhost:4200 Usage: Upload your code and dataset through the web interface at http://localhost:4200. If the dataset exceeds 50 GB, the tool automatically applies the chunk strategy. The resulting reproducibility manifest records the Zenodo DOI, chunk order, and MD5 checksum for each chunk.

Fields of Study & Subjects

Scientific Computing and Data Management Research Data Management Practices Academic Publishing and Open Access

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
25 March 2026
Language
EN