A practical approach to classifying code-snippet questions in small-scale educational repositories using LLM embeddings

Classifying code snippet-based questions is essential for teaching, preparing assessment materials, and supporting intelligent learning systems in programming education. Frequency-based encoding schemes, like the frequency-inverse document frequency (TF-IDF), struggle to capture the contextual semantics of code snippets in programming-related questions. This study uses the contextualized embeddings from a large language model, the text-embedding-3-large (TE3L), to investigate the effectiveness of contextualized embeddings in improving classification performance and the best classifier to work with. Using SQL certification questions as a case, the study examines the classification complexity reduction and performance improvements achieved by the TE3L compared to TF-IDF schemes. We also evaluate the classification performance of the single classifier, boosting, and stacking ensembles through the weighted macro-average F1 score, precision, and recall. The results demonstrate that the TE3L scheme significantly reduces classification complexity and improves performance compared to TF-IDF. Single classifiers, particularly the support vector machine with a linear kernel and the stochastic gradient descent classifiers, performed the best with the TE3L scheme and greatly outperformed the benchmark by 11% in the weighted macro-average F1 score. The boosting and stacking techniques did not enhance performance, likely due to class imbalance and limited training data. The findings demonstrate that TE3L encoding can streamline the implementation of automated question classification in programming education, making it more accessible to teachers without extensive technical expertise. This study is among the first to investigate the application of commercial contextualized embeddings for classifying code-snippet-based questions under practical educational constraints. The results provide valuable insights for developing automated classification systems in programming education, particularly for small-scale, discipline-specific question banks.