Multimodal emotion recognition for edge devices

This thesis presents a scalable, low-cost, and privacy-conscious framework for real-timeemotion recognition using Internet of Things (IoT) technologies and machine learning.Traditional systems often rely on high-cost, invasive hardware and are typically limited tocontrolled environments. In contrast, this research demonstrates that high-performanceemotion classification can be achieved using lightweight models and aordable sensorssuitable for deployment on edge devices.Three primary objectives were addressed: the evaluation of unimodal classiers acrosstext, speech, and thermal image data; the development of a reproducible data collectionframework using low-cost thermal imaging hardware; and the implementation of a realtimeMultimodal Emotion Recognition system optimised for embedded platforms. A novelthermal facial expression dataset was introduced, collected using an IoT-based thermalcamera across six emotions and five national backgrounds. For textual emotion recognition,traditional machine learning models combined with Term Frequency-Inverse DocumentFrequency features achieved up to 90% accuracy, outperforming transformer-basedmodels such as Distilled Bidirectional Encoder Representations from Transformers in botheciency and latency when deployed on edge hardware. In the thermal domain, a modiedVision Transformer achieved 96% classification accuracy, surpassing Residual Network inboth accuracy and generalisation across hybrid datasets. The final Multimodal EmotionRecognition system, combining thermal, speech, and text modalities using both featureleveland decision-level fusion, demonstrated robust real-time performance. Decision-levelfusion achieved up to 99% accuracy on structured datasets, with inference latency below0.8 seconds and throughput exceeding 4,200 samples per second on a Raspberry Pi 4B.These results confirm that accurate, real-time emotion recognition is achievable onaffordable hardware without compromising performance. The system developed in thisresearch offers a practical, deployable framework for emotion-aware computing in realworldapplications such as healthcare, education, and human-computer interaction.