Edge AI System Using Lightweight Semantic Voting to Detect Segment-Based Voice Scams

Real-time telecom scam detection is difficult without cloud AI, particularly for privacy-sensitive, low-resource environments. We developed a lightweight, offline voice scam detector using on-device audio segmentation, automatic speech recognition (ASR), and semantic similarity. Four detection strategies were implemented. We used Whisper ASR and DeepSeek to process 5 s speech chunks. An analysis of 120 synthetic and paraphrased Mandarin phone call dialogues reveals the A4 voting strategy’s superior performance in optimizing early detection and minimizing false positives, achieving an F1 score of 0.90, a 2.5% false positive rate, and a mean response time of under 4 s. The system is deployable on ESP32 for offline mobile inference. The proposed architecture provides a robust and scalable defense against threats targeting vulnerable user groups, such as older adults. It introduces a new method for real-time voice threat mitigation on devices through interpretable segment-level semantic analysis.