Re-examination of Stability in Generative AI-based Free Text Labeling

Read full text Summarise with VIRAT
Authors: Takahiro Sumiya, Takahiro Sumiya, Yuki Amano
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

While labeling free-form text with generative AI is useful, the inherent non-determinism of large language models (LLMs) can cause outputs to vary even when the temperature is set to 0. In this study, taking this non-determinism as a given, we organize methods for estimating confidence in labeling results using token-level generation probabilities (log probabilities; LogProbs). In addition, when LogProbs are unavailable, we examine an approximate way to assess confidence via a Monte Carlo–style approach that uses the output distribution obtained from repeated generations. This reframes labeling variability under generative AI as an indicator of ambiguity inherent in free-form responses and in evaluation procedures, and offers a perspective that can improve the reliability of educational data analysis.

Fields of Study & Subjects

Artificial Intelligence in Education Online Learning and Analytics Topic Modeling

Publication details

Journal / source
Institutional Repositories DataBase (IRDB)
Volume
2026-CE-183
Issue
9
Pages
1–6
Publication date
31 January 2026
Language
JA