From Sketch to BIM: A Framework for Automating the Conversion of Hand-Drawn Plans into 3D Models Using MLLMs and 3D Modeling Software

Building information modeling (BIM) has become essential in modern architectural, engineering, and construction (AEC) projects, enhancing efficiency by fostering collaboration among disciplines. However, in most instances, transforming 2D plans into detailed 3D models remains time-intensive and prone to many errors. This paper proposes an innovative framework to address these challenges by integrating pre-defined workflows with carefully crafted instructions and function-calling capabilities of multimodal large language models (MLLMs) with the Blender API. The framework uses structured prompts to guide MLLMs in performing pre-defined tasks, ensuring accurate processing of hand-drawn 2D building plans and automating their transformation into detailed 3D models. Findings of preliminary case studies confirm the framework's effectiveness in achieving this transformation. By blending advanced technology with practical design methods, the proposed framework ultimately minimizes manual labor and enables AEC professionals, as well as newcomers, to create 3D models more easily and efficiently.