Zero-shot vs. fine-tuned LLMs for customer complaint classification: a benchmark study on the CFPB dataset

This master's thesis investigates the performance and efficiency trade-offs between fine-tuned lightweight transformer models and zero-shot Large Language Models (LLMs) for classifying consumer financial complaints. Using a dataset from the Consumer Financial Protection Bureau (CFPB) covering the year 2025, the study benchmarks a fine-tuned DistilBERT model against state-of-the-art LLMs, including GPT-5.2, Claude Sonnet 4.5, and Grok 4.1, in a zero-shot setting. The research addresses data challenges by consolidating complaints into a robust six-category framework to mitigate class imbalance and semantic ambiguity. The empirical results demonstrate that the fine-tuned DistilBERT model achieved the highest predictive performance with an accuracy of 87.78% and a weighted F1-score of 0.8795, outperforming all zero-shot LLMs. Among the generative models, GPT-5.2 performed best with an accuracy of 87.32%, indicating that the gap between supervised and zero-shot approaches has significantly narrowed. However, the analysis highlights critical disparities in operational viability. While frontier reasoning models offered high accuracy, they incurred substantially higher costs compared to efficient models like Grok 4.1-fast and GPT-5-mini. The thesis concludes that while zero-shot LLMs are ideal for rapid prototyping and low-volume tasks, fine-tuned lightweight models remain the superior choice for high-throughput, cost-sensitive regulatory environments due to their economic scalability and consistent latency.