Reproducible Automated Pipeline for Evaluating LLMs on Standardized Exam Questions

This paper introduces an open-source evaluation pipeline designed to benchmark Large Language Models (LLMs) on standardized examination-style questions. The framework automates the end-to-end process: parsing examination PDFs, structuring content into a standardized JSON format, executing inference across various models, and calculating scores based on complex marking schemes. Critically, the pipeline moves beyond labor-intensive screenshot-based methods by employing a structured text-parsing architecture to ensure data integrity and evaluation precision. To demonstrate its utility, we evaluate state-of-the-art LLMs using the GATE CS 2025 examination. Results provide a comparative analysis of performance on technical reasoning tasks. The complete codebase and datasets are publicly released to facilitate reproducibility and provide a foundation for benchmarking other competitive examinations with minimal configuration.