Replication Package for "Position: Current Coding Benchmarks Measure Task Performance, not Coding Capabilities"

This replication package accompanies the paper "Position: Current Coding Benchmarks Measure Task Performance, not Coding Capabilities" and contains: Evaluation Results for 24 models across 3-4 benchmarks: Method Generation — Multi-turn code generation with real-world test feedback from the Django framework Code Completion — Single-turn completion tasks Program Repair — Bug fixing with test-driven feedback LiveCodeBench — Competitive programming evaluation Models Evaluated: 10 open-weight models (Qwen2.5-Coder, DeepSWE, Skywork-SWE, R2EGym, SWE-agent-LM, OpenHands) 6 LoRA fine-tuned configurations (32B and 7B variants) 8 proprietary API models (GPT-4.1, GPT-5, Claude Haiku/Sonnet/Opus 4.5, Gemini 2.5/3) Fine-tuned Checkpoints: LoRA adapter weights for Qwen2.5-Coder-32B-Instruct and Qwen2.5-Coder-7B-Instruct. Fine-tuned on method generation, completion, and program repair tasks Analysis Tools: Python utilities for loading and analyzing results Jupyter notebook for interactive exploration