LLM Token Estimation Benchmark Data

Read full text Summarise with VIRAT
Authors: Arnaud Leroy
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

This dataset presents benchmark results for token count estimation accuracy across 12 major large language models (LLMs) from leading providers, collected and published as part of the developer tooling research on mohitkhare.me. Accurate token estimation is a critical component of cost prediction, context window management, and prompt engineering for production LLM applications. Each row corresponds to a single model and includes the following fields: model name, provider, tokenizer family, average tokens per English word, estimation accuracy percentage (measured against exact tokenizer output on a standardized 10,000-sentence corpus), context window size, prompt overhead in tokens, encoding type (BPE or SentencePiece), and the date the benchmark was conducted. The 12 models benchmarked include GPT-4o and GPT-4 Turbo (OpenAI), Claude 3 Opus and Claude 3.5 Sonnet (Anthropic), Gemini 1.5 Pro (Google), Llama 3 70B (Meta), Mistral Large (Mistral), Command R+ (Cohere), Grok 3 (xAI), DeepSeek V3 (DeepSeek), Phi-3 Medium (Microsoft), and Qwen 2.5 72B (Alibaba). The benchmark corpus consists of diverse English text spanning news articles, technical documentation, conversational dialogue, and code snippets. Key findings include: BPE-based tokenizers consistently achieve higher estimation accuracy (96-97%) compared to SentencePiece variants (95-96%); Anthropic models exhibit the lowest tokens-per-word ratio (1.28) indicating efficient vocabulary coverage; and prompt overhead varies from 3 to 6 tokens depending on the model chat template. Detailed methodology and analysis are available on the developer blog at mohitkhare.me/blog.

Fields of Study & Subjects

Topic Modeling Natural Language Processing Techniques Software Engineering Research

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
30 March 2026
Language
EN
Document type
Dataset