Enhancing Candidate Generation in Recommendation Systems Through LLM-Powered Semantic Enrichment in a Distributed Environment

Effective candidate generation is a critical component of two-stage recommender systems; however, traditional methods such as Term Frequency–Inverse Document Frequency (TF-IDF) often fail to capture deep semantic context. This limitation leads to suboptimal recall rates, particularly for new or niche items—a challenge commonly referred to as the cold start problem—thereby degrading overall recommendation quality and user experience. This study proposes a semantically aware approach to improve the initial recall phase of recommendation pipelines. The methodology integrates Large Language Models (LLMs) into a distributed Apache Spark pipeline for large-scale content enrichment, generating 768-dimensional vector embeddings and concise, context-aware summaries for each content item. These enriched representations are indexed in Elasticsearch to enable efficient vector-based retrieval during candidate generation. Quantitative evaluation on a corpus of 143,000 Wikipedia articles demonstrates that the LLM-enriched method achieves a Recall@10 of 62%, representing a 37% relative improvement over the TF-IDF baseline (45%). When relevance is measured using only embedding-independent signals (category overlap and keyword similarity), the method still achieves a Recall@10 of 58%, confirming that gains are not an artifact of the evaluation metric. The resulting candidate pools exhibit improved semantic diversity and broader category coverage, delivering richer input for downstream ranking models.