Multimodal Llms For Image-Based Fabric Composition Identification

With e-commerce shaping the way people buy clothes today, correctly determining what fabrics are made of is now a key requirement. Conventional methods are often constrained by hardware dependencies (e.g, NIR sensors), limited accessibility and poor fabric classification. We propose a parameter-efficient multimodal system that uses instruction-tuned LLMs to determine the proportion of fibers directly from fabric images. We compare two different multimodal models. The first is Meta’s LLaMA 3.2-Vision 11B Instruct model, which builds on a separately trained image encoder and a vision adapter that integrates image-feature representations via cross-attention layers into a core language model. The second is Salesforce’s BLIP2, which connects a ViT-G/14 image encoder to an OPT language model. In both the models, a light-weight regression head is appended to the final pooled representations and trained alongside LoRA injected adapters targeting the query, key and value projections within the transformer layers. The LoRA configuration is implemented using the PEFT (Parameter-Efficient Fine-Tuning), reducing trainable parameters to a few million for efficient adaptation. Experiments span six fiber categories -cotton, wool, polyester, linen, silk, and other fibers using a custom dataset of labelled textile images. The BLIP2-based model achieved a mean absolute error of 13.40 percentage points, outperforming the LLaMA 3.2 model, which recorded 14.87 percentage points on the test set. The evaluation results validate the use of vision-language models for estimating fiber composition and point toward their potential adoption in both consumer and industrial applications.