CodeGen-3D: A Benchmark for Evaluating LLMs in Zero-Shot and Iterative 3D Modeling in Blender

Large Language Models (LLMs) that generate executable code are opening a new path to text-to-3D by converting natural language prompts into scripts for modeling software like Blender. However, there is a lack of evaluation for such methods currently. Existing benchmarks focus on direct mesh synthesis and overlook critical factors like code execution reliability and human-aligned quality. To address this gap, we introduce CodeGen-3D, a comprehensive benchmark and protocol for assessing 3D modeling via code generation. The benchmark consists of 100 diverse prompts derived from Objaverse with rich captions (Cap3D, DiffuRank, and Llama-based), a standardized Blender execution and six-view rendering pipeline, and a three-part evaluation protocol that assesses: (i) error rate (compilation and runtime failures), (ii) multi-view CLIP alignment (average and max) for semantic similarity, and (iii) a VLM-as-a-Judge pairwise preference task (using GPT-4o) for perceptual quality and prompt adherence. We report results for eight general-purpose LLMs, a specialized BlenderLLM, and an iterative self-correcting agent. Our analysis shows that specialized and iterative systems are substantially more reliable (3–4% failure rate vs. 21–92% for general LLMs) and obtain the highest GPT-4o preference rates (e.g., 33.3% win rate for the iterative agent), even when CLIP scores are similar. This gap between CLIP-based alignment and judge-based preferences underscores the need for feedback-driven, multi-signal evaluation, suggesting that practical text-to-3D systems should combine strong general LLMs with domain specialization or inexpensive self-refinement loops, rather than relying on single-pass code generation. CodeGen-3D establishes a practical, reproducible benchmark and offers strong baselines for future research at the intersection of LLMs, code generation, and 3D modeling.