Do Transformers Always Win? An Empirical Study of Semantic Embeddings for Short-Text E-commerce Reviews

While transformer-based embeddings like SentenceBERT have become the de facto standard for text representation, their performance on short, noisy industrial texts has received limited empirical scrutiny. This study compares TF-IDF, Word2Vec, and Sentence-BERT on clustering $\mathbf{100,000}$ Amazon product reviews. Our findings challenge prevailing assumptions: Word2Vec achieves superior clustering performance with a Silhouette score of 0.1828 versus SBERT’s 0.0401, representing a $356 \%$ improvement. We attribute this to insufficient text length for contextual modeling, domain mismatch between pre-training corpora and e-commerce reviews, and destabilizing variance in cluster centroids from contextualized representations. For topic modeling, Non-negative Matrix Factorization with Count Vectorizer achieves highest coherence ($\mathbf{C v}=\mathbf{0.5836}$), while Latent Dirichlet Allocation produces most balanced distributions. These results suggest classical methods offer compelling costperformance advantages for short industrial texts.