Tensor Low-Rank Orthogonal Compression for Convolutional Neural Networks

Dear Editor, The letter proposes a tensor low-rank orthogonal compression (TLOC) model for a convolutional neural network (CNN), which facilitates its efficient and highly-accurate low-rank representation. Model compression is crucial for deploying deep neural network (DNN) models on resource-constrained embedded devices. However, the typical approach based on prefine tuning is time-consuming, laborintensive and computationally expensive. Moreover, training a low-rank model from scratch presents significant challenges including substantial accuracy degradation and end-to-end training disruption. Motivated by these critical challenges, this letter presents the end-to-end TLOC model adopting the following three-fold ideas: 1) Utilizing the tucker-2 and truncated singular value decomposition (TSVD) decompositions to compress the weights of the Convolutional and fully connected (FC) layers, thus training a deep model in a low-rank format from scratch; 2) Incorporating an orthogonal regularization term into the compression process, thereby enhancing the TLOC model's representation ability to the structure of a CNN's weights; and 3) Incorporating a cyclic learning rate adjustment strategy into the training process for efficient models training. Experimental results on multiple deep networks and datasets demonstrate that the proposed TLOC model can establish highly accurate representation to a CNN without accuracy loss to its downstream tasks, thus outperforming benchmark and state-of-the-art models in implementing efficiency and accurate CNN compression.