"A Design-Theoretic Taxonomy for Deconstructing Engagement AI Audit Annotation Results Data Table"

Read full text Summarise with VIRAT
Authors: Chenhaoran Yu
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

"This dataset contains two complementary components for analyzing player engagement in interactive game systems: (1) a large-scale corpus of 14,717 Steam user reviews annotated with a custom 28-dimensional binary activation framework, and (2) a smaller reliability audit subset of 201 reviews (67 annotation tasks) designed to validate cross-model consistency.The smaller audit subset (N=201) consists of 67 targeted annotation tasks evaluated by three architecturally distinct large language models (Qwen-2.5-7B, Llama-3.1-8B, Gemma-2-9B) via a triangular consensus protocol. This component computes reliability metrics including Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) and triple-match rates to validate the robustness of the automated coding scheme, ensuring the main corpus annotations reflect consistent, design-theoretic signals rather than model-specific artifacts."

Fields of Study & Subjects

Sentiment Analysis and Opinion Mining Computational and Text Analysis Methods Digital Games and Media

Publication details

Journal / source
IEEE DataPort
Publication date
28 March 2026
Language
EN
Document type
Dataset