OmniSketch: Sketch-Guided Text-to-3D Generation With High-Fidelity Geometry and Texture

Sketch-based or text-based 3-D content generation, which rapidly converts abstract concepts into 3-D models, plays a vital role in accelerating digitalization. However, the sparse sketch, absent texture, and unaligned dataset pose considerable challenges for cross-modal synthesis. To address these issues, we introduce OmniSketch, a sketch-guided text-to-3D generation framework that decouples the complex sketch-to-3D task into a three-stage pipeline. First, we employ a conditional diffusion model to fuse structural constraints from the sketch with semantic descriptions, generating a high-quality intermediate reference view. Subsequently, a Rectified Flow model is utilized to produce a 3-D model with high-fidelity geometry. Finally, we apply a geometry-guided multiview image synthesis method to furnish the model with view-consistent, high-quality textural details. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches in geometric fidelity and texture consistency. Leveraging our framework, we also construct a multimodal sketch-to-3D dataset and an interactive platform to promote the application of digital content generation.