DivControl: Knowledge Diversion for Controllable Image Generation

Controllable diffusion models have shown great promise in visual generation by conditioning on diverse structural inputs, such as edges and depth. However, existing methods either require separate models for each control signal or rely on unified architectures with entangled representations, resulting in poor generalization and high computational cost when adapting to new conditions. We introduce \textbf{DivControl}, a decomposable pre-training framework for unified and efficient conditional generation. By applying knowledge diversion to the condition encoder, DivControl explicitly disentangles condition-agnostic semantics (learngenes) from condition-specific modulations (tailors) via structured SVD-based decomposition. A dynamic soft routing mechanism selects relevant tailors based on semantic task embeddings, enabling zero-shot generalization and parameter-efficient adaptation to novel control types. To further enhance feature alignment, we incorporate a representation alignment loss that anchors early condition features to high-level semantic embeddings. Extensive experiments on a 20-condition benchmark demonstrate that DivControl achieves state-of-the-art controllability with \textbf{300$\times$ less training cost} than prior methods. It improves FID by \textbf{2.33} on seen conditions and shows strong zero-shot and few-shot performance on unseen modalities, highlighting its scalability, modularity, and transferability in controllable diffusion.