Mr_OASO — Medical research Open Science Observatory

Mr_OASO — Medical research Open Science Observatory Automated extraction of 13 core open science practices (plus study-type classification) from published medical research articles, using locally run open-weights LLMs. Developed for the paper: Open science policies and core open science practices in major medical journals: development of an automatized tool based on a Large Language Model (Vinatier, Freyermuth, Millour, Le Bartz-Lyan, Buet, Lunel, Robillard, Naudet, Acher). Companion / independent validation study: 10.1101/2025.11.26.25341061. The extracted practices follow the core indicator set proposed by Cobey et al. (2023): pre-registration, data sharing, open-access publishing, code sharing, protocol sharing, statistical analysis plan sharing, reporting guideline use, publication within 1 year (RCTs), preprint availability, author contributions, conflicts of interest, ORCID reporting, and funding statement. Pipeline Document conversion — article PDFs are converted to TEI XML with GROBID 0.8.2; supplementary files are converted with PyMuPDF (fitz) / python-docx / plain readers. Chunk retrieval — the text is split into 256-token chunks (32-token overlap); for each criterion question, the top-8 chunks are retrieved with BM25. (A dense-retrieval alternative — BiomedBERT embeddings + ChromaDB — exists in the codebase but showed no benefit in pilots and is not used for reported results.) LLM-based classification — a local Llama model (via Ollama; Llama 3 8B / Llama 3 70B / Llama 3.3 70B, temperature 0.0) answers one constrained question per criterion (YES/NO or 0/1/2/3); answers are parsed into per-article labels. Repository layout Path Contents full_retrieve.py Main pipeline (one run per criterion per model) utils/ Retrievers (BM25, embeddings, Lancet XML), evaluation, reporting, config configs/models.json Model selection (answer_generation: llama3, llama3:70b, llama3.3) data/ Input CSVs: article lists + manual ground truth (details) results/ Aggregated classifications + comparisons to ground truth (details) output/ Run artefacts (GROBID TEI folders, per-run report folders) — mostly untracked eval.py, analysis_classification.py Evaluation against the ground truth rerun_summer/ Runbook + eskemm cluster job scripts for the paper's runs scripts/export_prompts.py Regenerates supplementary/prompts_supplement.md (all prompts, verbatim) supplementary/ Prompt supplement (basis for the paper's Supplement 3) notebooks/ Development/analysis archive — not maintained (details) changes.md Chronological development log (retrieval variants, prompt iterations) Dockerfile, entrypoint.sh, supervisord.conf Container image (GROBID + Ollama + Python env) mroaso_job.sh, inside_container.sh Slurm/Apptainer entry points (eskemm cluster) REPRODUCIBILITY.md Exact configuration of the paper's runs + archival checklist CITATION.cff Citation metadata Getting started With Docker (recommended) docker pull macher/mroaso # or: docker build . -t mroaso (~22 GB) The image bundles GROBID, Ollama, and the Python environment. On a Slurm cluster, see mroaso_job.sh (Apptainer). Interactive services can also be started manually: GROBID server + ollama serve (see inside_container.sh). Running the pipeline # 2020–2021 validation dataset ("summer") — CSV separator is "," python full_retrieve.py --base-csv data/base_documents_paths_fix1.csv --sep "," \ --grobid-output-path ./output/grobid_summer/ # 2022–2023 dataset ("202223") — CSV separator is ";" python full_retrieve.py --base-csv data/base_documents_202223_corrected.csv --sep ";" \ --grobid-output-path ./output/grobid202223/ Useful flags: --only-criteria "COI,Data_Sharing" to restrict criteria; --supplemental for the with-supplements condition (aggregation strategy via the SUPP_STRATEGY env var, default any-positive). The model is chosen in configs/models.json. ⚠️ The BM25 retriever ingests every .xml under --grobid-output-path; check the folder (ls /*.xml | wc -l) before a run — a wrong path produces silently wrong results. Evaluating python eval.py # or analysis_classification.py Outputs land in results/ (aggregated_classification_*.csv, comparison_results_*.csv) — naming conventions in results/README.md. Data availability The article PDFs (2020–2021 and 2022–2023 datasets) are not in the repository for copyright reasons; they are available on request. The manual ground-truth labels (DOIs + extracted practices) are in data/. Reproducing the paper Everything needed is indexed in REPRODUCIBILITY.md: exact model builds, GROBID version, retrieval settings, per-dataset commands, known traps, and the archival checklist (tag → GitLab release → Zenodo DOI → Software Heritage SWHID). All LLM prompts as executed (including abandoned variants) are in supplementary/prompts_supplement.md. Citation See CITATION.cff. License: to be decided before release. Funding Developed within the OSIRIS project (Open Science to Increase Reproducibility in Science), European Union's Horizon Europe Research and Innovation Programme, Grant Agreement No. 101094725.