TrendSpotter: A Graph-based Real-Time Topic Detection System for Context-Aware Trend Discovery in Social Media

Real-time data from social media sites like Twitter, Instagram and Reddit is constantly being produced via users. Because there is no pre-defined set of keywords, finding trending topics by using keyword searches is inefficient. We developed a trend-detecting model called TrendSpotter, which employs graph and processing techniques of Natural Language Processing (NLP) to create semantic-co-occurrence graphs and uses community detection to group related posts together. It then measures centrality to determine the most important/ relevant trends.TrendSpotter can be used instead of simple keyword-based search methods because it captures contextual information about what is currently trending. Experimental results demonstrate TrendSpotter’s ability to detect trends quickly relative to other systems with similar anomaly-detection capabilities. The experimental results demonstrated that TrendSpotter was able to execute 27 times quicker than the BERT Baseline (0.067 seconds vs 1.858 seconds). TrendSpotter also maintained competitive anomaly-detection accuracy (Z = 1.65 vs Z = 1.41) and created more defined topic groups than did the BERT Baseline.