A Leakage-Audited, Regime-Robust Protocol for Evaluating Stock-Movement Prediction From Affective Text

# A Leakage-Audited, Regime-Robust Protocol for Evaluating Stock-Movement Prediction From Affective Text Code and reproducible pipeline for the paper *"A Leakage-Audited, Regime-Robust Protocol for Evaluating Stock-Movement Prediction From Affective Text"* (A. El Haddad, S. Achchab, Y. Lahrichi). Working paper, under review. ## Overview This project tests whether affective features—FinBERT sentiment, sarcasm, six emotion dimensions, and emoji scores—extracted from roughly 3.06 million cleaned social-media and news messages (from approximately 3.4 million collected) improve machine-learning prediction of the daily and ten-day movement of seven U.S. mega-cap stocks (AAPL, MSFT, NVDA, GOOGL, AMZN, META, TSLA).Under a leakage-audited, regime-stratified evaluation, the apparent gains do **not** survive: sentiment co-moves with the market regime rather than anticipating it. This notebook implements the full evaluation protocol so the analysis can be reproduced and reused on other temporal, class-imbalanced affective-text problems. ## Method / pipeline 1. **Multi-source data collection** (see *Data sources* below)2. **NLP scoring:** FinBERT sentiment, sarcasm (irony), six emotion dimensions, emoji sentiment3. **Feature engineering:** 49 features per stock-day (31 technical + 18 NLP), all lagged one day4. **Leakage audit** + strict chronological split (train Raw third-party text data are under their respective providers' licenses and are **not** redistributed here. The derived daily features used in the paper are available from the corresponding author upon reasonable request. ## Reproduce ```bashpip install -r requirements.txt# then run Pipeline.ipynb cells in order (data collection -> NLP scoring -># features -> leakage audit -> training -> evaluation/robustness)```Requirements: Python 3.10+, and the packages listed in `requirements.txt`. ## Canonical metrics The numbers reported in the paper are those saved to disk in `sentiment_pipeline/models/*.json` (test_metrics.json, ablation.json, backtest_summary.json) and `RESULTS.txt`; **these files are the single source of truth.** Some inline cell outputs retained from earlier development runs may differ; where they disagree, the saved artifacts and the Canonical Results cells at the end of the notebook are authoritative and match the manuscript and figures. ## Key results | Test | Result ||---|---|| Daily direction (1 day) | No better than chance (AUC ~0.51, not significant) || Ten-day direction (best model, 2023 bull) | AUC 0.644 (XGBoost) — never beats the majority-class baseline (0.683) || Sentiment ablation lift (10-day) | +0.71% (2023 bull) / -0.63% (2022 bear) — within the placebo range || Noise placebo (10-day) | -1.31% (2023) / -1.53% (2022): real lift indistinguishable from noise || Bear regime (2022) | LightGBM AUC 0.479 (anti-predictive) || Lag structure | \|r\| ~0.196 same-day vs 0.030 next-day -> contemporaneous, not predictive || Random vs chronological split | Validation AUC 0.828 vs 0.542 (gap 0.286 = look-ahead leakage) || Backtest (2023) | Strategy +3.28% vs market +112.33% -> alpha -109.05% |**Conclusion:** sentiment follows the market regime; it does not lead it. ## Citation El Haddad, A., Achchab, S., & Lahrichi, Y. (2026). A Leakage-Audited, Regime-Robust Protocol for Evaluating Stock-Movement Prediction From Affective Text [Software]. Zenodo. https://doi.org/10.5281/zenodo.20945551 ## LicenseCode: MIT (see `LICENSE`). Data: under the original providers' licenses.## ContactAbdelkhaleq El Haddad — abdelkhaleq_elhaddad@um5.ac.maENSIAS, Mohammed V University, Rabat, Morocco