The impact of the size of training sets on quality of automatic short answers grading

The paper investigates the impact of training set size on the quality of automatic short answers grading, formulated as a classification task. The impact was evaluated using a method based on measuring the similarity between the assessed answer and a given reference answer, calculated via embedding vectors, in combination with a logistic regression classifier. Experiments were conducted on corpora of answers to questions in computer science, history, and software development using Qt framework. The sizes of the corpora were 547, 522, and 931 answers, respectively. Two experiments were conducted during the study. In the first experiment, the change in classification quality was assessed as the training set size was reduced. It showed that when the binary classification is utilized (an answer can be either correct or incorrect), reducing the size of the training set leads to a smaller decline in quality compared to ternary classification (which includes a class of partially correct answers). In the second experiment, the possibility of improving classification quality by expanding small‑sized training sets through data augmentation was investigated. It demonstrated that augmentation performed using the DeepSeek generative model can significantly improve results in several cases, which is important for practical applications under data scarcity conditions. Additionally, the experiments revealed that when different language models are used to generate embeddings, the magnitude of change in classification quality with varying training set sizes can differ significantly. Specifically, using certain models — such as rubert‑tiny2 and MiniLM‑L12‑v2 — to produce embeddings yields more stable results than using other models.