TarikYil/turkish-llm-eval: v1.0.0 — Turkish LLM Benchmark Suite
Abstract
Turkish LLM Benchmark Suite v1.0.0 Initial release of the evaluation framework accompanying our paper. What's Included Evaluation pipeline for 7 open-source Turkish LLMs 6 benchmark implementations: MMLU-TR, ARC-TR, HellaSwag-TR, GSM8K-TR, TruthfulQA-TR, HalluVerse-M3 4-bit / 8-bit quantization support via BitsAndBytes Automatic chat template detection for Qwen2.5 and Gemma-2 architectures JSON + CSV result export Evaluated Models CosmosGemma-9B, Trendyol-7B v0.1 & v4.1, WiroAI-9B & 8B, KoçDigital-8B, Kumru-2B Authors Muhammet Tarık Yılmaz Yusuf Çakır
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Multimodal Machine Learning Applications
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 4 March 2026
- Language
- EN