FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries

Advancements in large multimodal models (LMMs) for text and image understanding have provided a new paradigm for semiconductor intelligent manufacturing. However, due to the knowledge-intensive nature of the integrated circuit (IC) domain, their effectiveness in real-world applications is limited. To address this, we propose a customized model, FabGPT, to enhance the wafer image detection and domain knowledge querying in real manufacturing environments. The model includes a modal enhancement stage for optimized wafer feature representation, a detection stage for identifying tiny wafer defects, and a question-answering (Q&A) stage that combines modulation modules with an interactive corpus training strategy, improving the performance of LMM in semiconductor manufacturing scenarios. To further improve FabGPT’s performance in scenarios with limited annotations, we introduce an unsupervised learning paradigm, utilizing selective storage and matching strategies to achieve pixel-level defect localization and querying across various scenarios. Additionally, in the Q&A stage, we upgrade a multi-objective modulation module and enrich a specialized corpus to systematically assess the relationships between modalities, seamlessly integrating IC knowledge into the pre-trained model. This improves the accuracy of root cause analysis and addresses the “modality bias”. Extensive experiments demonstrate that FabGPT excels at pixel-level defect detection without the need for defect labels and surpasses other large models in expert-level process Q&A tasks, showcasing its potential as a transformative tool in IC manufacturing.