A Hardware-Aware Efficient LLM for Multimedia Understanding via Dynamic Rank Adaptation and Structured Pruning

Large Language Models (LLMs) have been applied in multiple fields due to their language understanding, multi-modal interaction, and logical reasoning capabilities. This research develops a domain-specific LLM for multimedia content understanding (Media-LLM), which improves performance and efficiency through systematic optimization of dataset construction, parameter-efficient fine-tuning, structured pruning, and hardware acceleration. Initially, we built a high-quality multi-modal dataset for Media-LLM fine-tuning. This dataset integrates cross-modal alignment information among images, videos, and texts. By using this dataset, we fine-tune LLM by a parameter-efficient fine-tuning (PEFT) method based on dynamic rank adjustment which achieved a balance between training efficiency and model performance by dynamically allocating the sparsity of matrix ranks. Furthermore, to reduce computational costs, a structured pruning strategy based on the maximum-flow minimum-cut algorithm, which models parameter importance as a network flow problem to achieve hardware-friendly sparsification, was designed. Finally, the LLM inference speed was increased by optimizing the parallel computing architecture of GPU tensor cores. Experimental results show that our algorithm achieves performance improvements of 7.83 percentage points in task accuracy (78.45% vs. 70.62% for Llama-3.1-8B as shown in Table III), 27.8% in perplexity reduction (11.87 vs. 15.58 for prompt-tuned Llama-2-7B as demonstrated in Table VIII), and 2.53 percentage points in ACC7 metric (41.22% vs. 38.69% for Llama-3.1-8B as evidenced in Table I) compared to the strongest baselines in corresponding benchmark tests, excelling in terms of fine-tuning convergence time, efficiency, and final model inference speed.