From Engineering Drawings to Assembly Instructions: A Vision and Language Model Approach

Engineering drawings such as CAD draft sheets are widely used in manufacturing to document product structure, part geometry, and dimensional specifications. While these doc- uments contain valuable information, they are not typically organized to support step-by-step assembly tasks, which can present challenges for non-expert technicians during installa- tion, maintenance, or repair. This paper presents a system that automatically generates structured and human-readable assembly instructions from CAD drafts by combining a vi- sion model, an OCR model, and a language model. The vision model, trained on a constructed synthetic dataset, was able to detect mechanical components with an average precision score of 95.2% on real CAD sheets, while the OCR model suc- cessfully extracted dimensional information. These outputs, together with existing description text, were processed by a language model to produce clear and interpretable assembly steps. A synthetic dataset was used to train the vision model, addressing the lack of publicly available CAD annotations. The results demonstrate that the proposed system improves the interpretability and usability of engineering documentation in assembly-related tasks.