A survey of latent factorization of tensor-based model compression: Algorithms, toolboxes and future directions

Modern neural networks (NNs), while effective at learning representations from given samples and handling downstream pattern recognition tasks, typically contain tens to hundreds of millions of parameters. The growth in NN size motivates ongoing research on effective network compression with the purpose of reducing the computational burden without significantly sacrificing the model performance. It is especially critical when deploying NNs on resource-constrained devices where the computation and storage efficiency are highly concerned. A promising and currently popular solution to model compression is to replace the NN weight matrix with its low-rank tensor approximation, i.e., implementing an efficient latent factorization of tensors (LFT) process on the NNs parameters. Based on thorough investigations into the state-of-the-art LFT-based model compression methods, this survey 1) provides a comprehensive review on the latest research progress of LFT-based model compression methods on various NNs (e.g., Convolutional NNs, Recurrent NNs, and Transformers); 2) summarizes a number of widely-used LFT toolboxes; 3) evaluates LFT methods for model compression on a variety of main-stream NN backbones; and 4) discusses the development trend of LFT-based model compression technique. This survey aims to provide a systematic and comprehensive survey concerning the LFT-based model compression methods to artificial intelligence researchers and engineers, thereby promoting further research development in this crucial field.