Re-examination of Stability in Generative AI-based Free Text Labeling
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
While labeling free-form text with generative AI is useful, the inherent non-determinism of large language models (LLMs) can cause outputs to vary even when the temperature is set to 0. In this study, taking this non-determinism as a given, we organize methods for estimating confidence in labeling results using token-level generation probabilities (log probabilities; LogProbs). In addition, when LogProbs are unavailable, we examine an approximate way to assess confidence via a Monte Carlo–style approach that uses the output distribution obtained from repeated generations. This reframes labeling variability under generative AI as an indicator of ambiguity inherent in free-form responses and in evaluation procedures, and offers a perspective that can improve the reliability of educational data analysis.
Fields of Study & Subjects
Artificial Intelligence in Education
Online Learning and Analytics
Topic Modeling
Publication details
- Journal / source
- Institutional Repositories DataBase (IRDB)
- Volume
- 2026-CE-183
- Issue
- 9
- Pages
- 1–6
- Publication date
- 31 January 2026
- Language
- JA