Developer Utilities Reference Data: Token Costs, Model Pricing, and Text Processing Benchmarks

Read full text Summarise with VIRAT
Authors: Mohit Khare
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Reference datasets compiled by Mohit Khare for developer utilities and tooling. This dataset provides three categories of structured data useful for software engineers working with large language models and text processing systems. Contents include: (1) Comparative pricing data for major LLM APIs including Anthropic Claude, OpenAI GPT, Google Gemini, Meta Llama, Mistral, and DeepSeek, with per-million-token costs and context window sizes. (2) Empirical token estimation benchmarks measuring character-to-token and word-to-token ratios across tokenizers, plus language-specific multipliers for 10 languages. (3) Performance benchmarks for common text processing operations (tokenization, NER, regex, JSON serialization) across popular Python libraries. Related resources: mohitkhare.me - Developer portfolio and utilities Blog - Technical writing on development topics

Fields of Study & Subjects

Natural Language Processing Techniques Topic Modeling Artificial Intelligence Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
28 March 2026
Language
EN
Document type
Dataset