UI-Native One-Shot Benchmarking for Mathematical Reasoning in Chat-Based LLM Systems

Zenodo (CERN European Organization for Nuclear Research) 2026 Preprint Open access

Read full text
Authors: Aaditya Paudel, Paru Acharya
DOI: 10.5281/zenodo.18216958
Cited by: 0

Abstract

This preprint presents a UI-native benchmark for evaluating mathematical reasoning in chat-based large language model (LLM) systems. The benchmark is derived from the published puzzle codex Vault of Echoes: Volume I and evaluates single-shot (pass@1) performance under strict output directives across contemporary chat products. We report accuracy, uncertainty estimates, and UI-layer failure modes, and release a machine-readable evaluation ledger to support reproducibility.

Fields of Study & Subjects

Topic Modeling Natural Language Processing Techniques Artificial Intelligence Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
12 January 2026
Language
EN
Document type
Preprint