Content-routed hybrid extraction from free-text pathology reports in Brazilian Portuguese: a human-anchored internal and external validation study

Read full text Summarise with VIRAT
Authors: RAFAEL DE NEGREIROS BOTAN
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

STUDY DESIGN: Diagnostic accuracy study (STARD-AI compliant). Retrospective analysis of free-text pathology reports in Brazilian Portuguese. SOURCE COHORT: 848,987 pathology reports (2015-2025) from a private pathology laboratory (Lâmina Lab) in Brasília, Brazil. Reports include anatomopathological (AP), immunohistochemistry (IHC), cytology, and molecular biology exams across breast, gastric, thyroid, cervical, and other domains. SYSTEM UNDER VALIDATION: Hybrid AI extraction system with 4 components: 1. Content-based routing (validates IHC reports by conclusion content, not specimen metadata — addressing 27% misrouting rate found in development) 2. Domain-selective extraction: regex + LLM for breast AP/IHC; regex-only for gastric; regex + LLM-subtypes for thyroid; regex-first for cervical 3. Ontological canonicalization (behavior_class replacing benign/malignant; dual HER2/Ki-67 representation) 4. Reconciliation with discordance flagging Models: Claude Sonnet 4 (primary extractor), Claude Opus 4 (secondary dual reader — does not correct primary output). DATASETS: - Development set: 145 reports (used for prompt iteration) - Internal benchmark: 656 reports (used for pipeline decisions — NOT test data) - Frozen holdout: 980 reports (seed=51, zero overlap with above) — PRIMARY EVALUATION - Challenge set: 100 reports (separate stress test, not included in success thresholds) - External validation: ~400 reports from independent laboratory (Diagnose) HOLDOUT COMPOSITION: - Breast AP: 200 (80 biopsy carcinoma, 60 surgical, 60 benign) - Breast IHC: 200 (20 HER2 3+, 30 HER2 2+, 30 triple-negative, 80 luminal, 40 non-breast IHC) - Gastric: 180 (60 OLGA 0, 50 OLGA I, 40 OLGA II, 15 H.pylori+, 15 dysplasia) - Thyroid Bethesda: 200 (25 Cat I, 25 Cat II, 40 Cat III, 40 Cat IV, 35 Cat V, 35 Cat VI) - Cervical: 200 (80 NILM, 40 ASC-US, 20 ASC-H, 25 LSIL, 25 HSIL, 10 AGC) REFERENCE STANDARD: - Primary: Pathologist 1 (P1) — blinded annotation of ALL 980 holdout reports - Reliability: Pathologist 2 (P2) — blinded annotation of 150 reports (30 per domain, stratified) - Adjudication: Pathologist 3 (P3) — resolves P1↔P2 discordances in the 150 subsample - External: P1 reviews 100% of ~400 Diagnose reports PRIMARY ENDPOINTS (per domain): - Breast AP: behavior_class, tipo_hist_canonical, margem_status, invasive_malignancy - Breast IHC: re_status, rp_status, her2_status - Gastric: h_pylori, olga, displasia - Thyroid: bethesda_category - Cervical: resultado_bethesda, adequabilidade SUCCESS THRESHOLDS: - F1/macro-F1 ≥ 0.90 for primary categorical/binary variables per domain - Weighted kappa ≥ 0.85 for ordinal variables - ICC ≥ 0.85 for numeric variables - Inter-rater reliability (P1↔P2) kappa ≥ 0.80 - Study passes if ≥ 4 of 5 domains meet all primary thresholds ANALYSIS PLAN: - Metrics computed after ontological canonicalization; raw agreement reported in supplement - CIs via stratified bootstrap (2,000 iterations) - Secondary analyses: inter-AI agreement (Sonnet↔Opus), performance by pathologist/period, challenge set FROZEN PIPELINE: Version v5.2.1 — No further tuning of prompts, regex, routing, ontology, or canonicalization rules after this registration. Any post-hoc deviations will be documented and reported. ETHICS: Protocol submitted to CEP/FEPECS (12/02/2026). Broad scope. Waiver of informed consent. DATA AVAILABILITY: Raw reports contain PHI and cannot be shared. Will provide: prompts, regex code, canonicalization rules, anonymized outputs, analysis code. REGISTRY DATE: 2026-03-21

Fields of Study & Subjects

Biomedical Text Mining and Ontologies Artificial Intelligence in Healthcare and Education Machine Learning in Healthcare

Publication details

Publication date
1 January 2026
Language
EN
Document type
Preprint