Developer Utilities Reference Data: Token Costs, Model Pricing, and Text Processing Benchmarks
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Reference datasets compiled by Mohit Khare for developer utilities and tooling. This dataset provides three categories of structured data useful for software engineers working with large language models and text processing systems. Contents include: (1) Comparative pricing data for major LLM APIs including Anthropic Claude, OpenAI GPT, Google Gemini, Meta Llama, Mistral, and DeepSeek, with per-million-token costs and context window sizes. (2) Empirical token estimation benchmarks measuring character-to-token and word-to-token ratios across tokenizers, plus language-specific multipliers for 10 languages. (3) Performance benchmarks for common text processing operations (tokenization, NER, regex, JSON serialization) across popular Python libraries. Related resources: mohitkhare.me - Developer portfolio and utilities Blog - Technical writing on development topics
Fields of Study & Subjects
Natural Language Processing Techniques
Topic Modeling
Artificial Intelligence Applications
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 28 March 2026
- Language
- EN
- Document type
- Dataset