DiffBench Meets DiffAgent: End-to-End LLM-Driven Diffusion Acceleration Code Generation

Diffusion models have achieved remarkable success in image and video generation. However, their inherent multi-step inference process results in substantial computational overhead during inference, posing significant challenges for real-world deployment. Therefore, accelerating diffusion models is of great practical importance. Existing acceleration techniques include model quantization, model pruning, sampler optimization, step reduction, and compilation-level optimization. Determining how to effectively combine multiple acceleration techniques to achieve optimal performance for a given diffusion model remains a major challenge for engineers. To address this, we propose the Diffusion Optimization Agent, an automated framework designed to generate the optimal acceleration strategy and corresponding code for any given diffusion model. Additionally, we introduce DiffBench, a comprehensive benchmark covering diverse diffusion model pipelines, combinations of optimization techniques, and acceleration tasks. This paper presents a detailed description of the DiffBench construction process and the design principles of the Diffusion Optimization Agent. Extensive experiments demonstrate that our agent significantly outperforms current state-of-the-art large language models (LLMs) in generating effective acceleration strategies for diffusion models.