SURSY data: Textual data for syndromic surveillance

Datasets produced in the context of the SURSY project to conduct syndromic surveillance for plant health (monitoring potentially new diseases (i.e. disease X) and the development of already existing diseases on new host plants). ====== Phase 1: Articles collected with PADI-web labeled by thematic categories: SV: Plant health (Santé Végétale) SA: Animal health (Santé Animale) SP: Public health (Santé Publique) Phase 2: Articles collected with PADI-web labeled by syndromic categories: VS: Syndromic surveillance (veille Syndromique) NVS: Non syndromic surveillance (Non Veille Syndromique) ====== Phase 1 Datasets: Title, Text, Thematic (SV/SA/SP), Disease, Text_without_html, Clean_text - P1-UMAP-KMeans – Articles vectorized with TF-IDF (max 10,000 features), reduced to 3D using UMAP, clustered with K-means, and selected around cluster centroids to ensure thematic balance. CSV file: data_selected_with_umap_kmeans_phase1.csv - P1-DiseaseBalanced – Balanced selection by disease within SV, then proportional sampling in SA and SP, with undersampling to maintain disease diversity. CSV file: data_selected_by_disease_phase1.csv ====== Phase 2 Datasets Text, Tokens_clean, Text_clean, Type_article (VS/NVS), Thematic (SA/SV) - P2+P1-NoDiseaseNamesRatio – Phase 2 data (with disease names) combined with a portion of Phase 1 data (without disease names) to achieve a 1:10 VS/NVS ratio. Phase 1 articles are considered NVS. CSV file: data_final_phase2.csv The P2+P1-NoDiseaseNamesRatio dataset gave the best performance and was retained as the main Phase 2 dataset Other data of Phase 2 (private): - P2-Only-WithDiseaseNames – Articles collected exclusively in Phase 2, containing disease names. CSV file: data_phase2_only_with_disease_names.csv - P2+P1-WithDiseaseNamesRatio – Phase 2 data (with disease names) combined with a portion of Phase 1 data (also with disease names) to achieve the same 1:10 ratio. CSV file: p2_with_p1_both_disease_names_public.csv - P2+P1-WithDiseaseNames-Cleaned – Data from version 3 with all disease names removed. CSV file: p2_with_p1_both_no_disease_names_public.csv