TURTLEBENCH: Evaluating Top Language Models via Real-World Yes/No Puzzles

Existing benchmarks for Large Language Models (LLMs) are often static, knowledge-dependent, or costly, hindering the reliable evaluation of reasoning in dynamic user interactions. To address this, we propose TurtleBench, a novel benchmark derived from real user guesses in an online "Turtle Soup Puzzle" game. This approach yields a dynamic dataset that isolates logical reasoning from background knowledge, mitigating risks of data contamination and model cheating. TurtleBench contains 1,532 annotated user guesses, on which we evaluated nine leading LLMs. Our results show that OpenAI's o1 series models lag behind other top competitors. We hypothesize this stems from limitations in their latent reasoning, such as simplistic Chain-of-Thought (CoT) processes and noise from longer reasoning paths. The dataset and code are publicly available at https://github.com/mazzzystar/TurtleBench.