Deceptive Simplicity: How high F1 Scores Obscure Artifact-Driven Performance in Indic Fake News Detection

High F1 scores in Indic misinformation classification have suggested significant progress in its detection. However, we challenge this narrative by demonstrating that such performance is reflective of dataset artifacts rather than a robust semantic understanding of the text. Using a BoW, TF-IDF & Logistic regression baseline that converges with transformer performance in our preferred languages 1 , we provide empirical evidence that classification exploits surface patterns. When evaluated on a noisier, intentionally chosen out-of-distribution dataset: BharatFakeNewsKosh, both BoW and transformers degrade simultaneously, confirming genuine task difficulty rather than an architectural limitation. Crosslingual transfer experiments from Patil to Bharat reveal a catastrophic F1 drop proving that learned patterns are dataset-specific and non-generalizable. These findings attempt to question the benchmark narrative of the task being 'solved' and argue for data-centric evaluation protocols emphasizing out-of-distribution robustness.