Journal of South China University of Technology(Natural Science) >

2024 , Vol. 52 >Issue 7: 29 - 38

DOI: https://doi.org/10.12141/j.issn.1000-565X.230617

Electronics, Communication & Automation Technology

A TT-Tucker Decomposition-Based LC Convolutional Neural Network Compression Method Without Pre-Training

  • LIU Weirong ,
  • ZHANG Zhiqiang ,
  • ZHANG Ning ,
  • MENG Jiahao ,
  • ZHANG Min ,
  • LIU Jie
Expand
  • College of Electrical and Information Engineering，Lanzhou University of Technology，Lanzhou 730050，Gansu，China
刘微容（1976—），男，博士，教授，博士生导师，主要从事工业过程先进控制理论与应用、图像处理与模式识别研究。E-mail： liuwr@lut.edu.cn

Received date: 2023-10-07

  Online published: 2024-02-02

Supported by

the National Natural Science Foundation of China(62261032);the Natural Science Foundation of Gansu Province(22JR5RA272);the Key Talent Project of Gansu Province

Fold

Abstract

Tensor training (TT) decomposition and Tucker decomposition are two effective compression methods for convolutional neural networks. However, TT and Tucker decomposition face the problems of spatial structure information loss and high computational complexity respectively. To solve the above problems, this paper considered the information retention rate and resource occupancy of the network structure and proposed a LC convolutional neural network compressed method (TT-LC) without pre-training based on TT-Tucker decomposition, adopting the learning-compression (LC) algorithm constraint compression framework. The TT-LC method includes two parts: learning step and compression step. The learning step didn’t not need the pre-training process, and adopted the exponential cyclic learning rate method to improve the training accuracy. In the compression step, this paper selected the global optimal rank according to the advantages of TT and Tucker decomposition and the characteristics of Bayes rule, and used empirically variable Bayesian matrix factorization (EVBMF) and Bayesian optimization (BayesOpt) to select reasonable ranks to guide tensor decomposition. The TT-LC method was used to compress the trained model. TT-LC method not only reduces the loss rate of spatial structure information and computational complexity, but also solves the problem that the unreasonable rank selection of the tensor leads to the significant decrease in model accuracy. It can realize the double Bayesian rank selection and double compression of the model, and obtains the optimal compression model. Finally, experiments were carried out on CIFAR10 and CIFAR100 datasets using ResNets and VGG networks. The results show that for ResNet32 network, compared with the benchmark method, the proposed method achieved a compression rate of parameter quantity of 69.6% and a floating point computation compression rate of 66.7% with the accuracy of 92.22%.

Key words： convolutional neural network; network compression; tensor decomposition; Bayesian optimization; constrained compression

Cite this article

LIU Weirong , ZHANG Zhiqiang , ZHANG Ning , MENG Jiahao , ZHANG Min , LIU Jie . A TT-Tucker Decomposition-Based LC Convolutional Neural Network Compression Method Without Pre-Training[J]. Journal of South China University of Technology(Natural Science), 2024 , 52(7) : 29 -38 . DOI: 10.12141/j.issn.1000-565X.230617

References

1 PUNYANI P， GUPTA R， KUMAR A ．Neural networks for facial age estimation：a survey on recent advances［J］．Artificial Intelligence Review，2020，53（5）：3299-3347.
2 SU N， CHEN X， GUAN J，et al ．Maritime target detection based on radar graph data and graph convolutional network［J］．IEEE Geoscience and Remote Sensing Letters，2022，19：4019705/1-5.
3 CONG S， ZHOU Y ．A review of convolutional neural network architectures and their optimizations［J］．Artificial Intelligence Review，2022，56（3）：1905-1969.
4 高晗，田育龙，许封元，等 ．深度学习模型压缩与加速综述［J］．软件学报，2021，32（1）：68-92.
  GAO Han， TIAN Yu-long， XU Feng-yuan，et al ．Survey of deep learning model compression and acceleration［J］．Journal of Software，2021，32（1）：68-92.
5 IDELBAYEV Y， CARREIRA-PERPI?áN M á ．Low-rank compression of neural nets：learning the rank of each layer［C］∥ Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：8046-8056.
6 魏钰轩，陈莹 ．基于自适应层信息熵的卷积神经网络压缩［J］．电子学报，2022，50（10）：2398-2408.
  WEI Yu-xuan， CHEN Ying ．Convolutional neural network compression based on adaptive layer entropy［J］．Acta Electronica Sinica，2022，50（10）：2398-2408.
7 WU J， CONG L， WANG Y，et al ．Quantized convolutional neural networks for mobile devices［C］∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2016：4820-4828.
8 JI M， SHIN S， HWANG S，et al ．Refine myself by teaching myself：feature refinement via self-knowledge distillation［C］∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：10659-10668.
9 ELSKEN T， METZEN J H， HUTTER F ．Neural architecture search：a survey［J］．Journal of Machine Learning Research，2019，20：1997-2017.
10 LEBEDEV V， GANIN Y， RAKHUBA M，et al ．Speeding-up convolutional neural networks using fine-tuned CP-decomposition［C］∥ Proceedings of the 3rd International Conference on Learning Representations.San Diego：OpenReview.net，2015：1-11.
11 NOVIKOV A， PODOPRIKHIN D， OSOKIN A，et al ．Tensorizing neural networks［C］∥ Proceedings of the 28th International Conference on Neural Information Processing Systems．Montreal：ACM，2015：442-450.
12 WANG W， SUN Y， ERIKSSON B，et al ．Wide compression：tensor ring nets［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：9329-9338.
13 KIM Y D， PARK E， YOO S，et al ．Compression of deep convolutional neural networks for fast and low power mobile applications［C］∥ Proceedings of the 4th International Conference on Learning Representations.San Juan：OpenReview.net，2016：576-584.
14 NAKAJIMA S， SUGIYAMA M， BABACAN S D，et al ．Global analytic solution of fully-observed variational Bayesian matrix factorization［J］．Journal of Machine Learning Research，2013，14（1）：1-37.
15 KIM T， LEE J， CHOE Y ．Bayesian optimization-based global optimal rank selection for compression of convolutional neural networks［J］．IEEE Access，2020，8：17605-17618.
16 KOLDA T G， BADER B W ．Tensor decompositions and applications［J］．SIAM Review，2009，51（3）：455-500.
17 CHENG Z， LI B， FAN Y，et al ．A novel rank selection scheme in tensor ring decomposition based on reinforcement learning for deep neural networks［C］∥ Proceedings of 2020 IEEE International Conference on Acoustics，Speech and Signal Processing．Barcelona：IEEE，2020：3292-3296.
18 LI N， PAN Y， CHEN Y，et al ．Heuristic rank selection with progressively searching tensor ring network［J］．Complex & Intelligent Systems，2022，8：771-785.
19 BESAG J ．On the statistical-analysis of dirty pictures［J］．Journal of the Royal Statistical Society Series B：Methodological，1986，48（3）：259-302.
20 CAI G Y， LI J H， LIU X X，et al ．Learning and compressing：low-rank matrix factorization for deep neural network compression［J］．Applied Sciences，2023，13：2704/1-22.
21 XU Y， LI Y， ZHANG S，et al ．Traned rank pruning for efficient deep neural networks［C］∥ Proceedings of 2019 the Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition．Vancouver：IEEE，2019：14-17.
22 HE Y， ZHANG X， SUN J ．Channel pruning for accele-rating very deep neural networks［C］∥ Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：1398-1406.
23 HE Y， LIN J， LIU Z，et al ．AMC：autoML for model compression and acceleration on mobile devices［C］∥ Proceedings of the 15th European Conference on Computer Vision．Munich：Springer，2018：815-832.
24 LI H， KADAV A， DURDANOVIC I，et al ．Pruning filters for efficient ConvNets［C］∥ Proceedings of the 5th International Conference on Learning Representations．Toulon：OpenReview.net，2017：1-13.
25 YU R， LI A， CHEN C F，et al ．NISP：pruning networks using neuron importance score propagation［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：9194-9203.
26 KIM H， KHAN M U K， C-M KYUNG ．Efficient neural network compression［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：12561-12569.
27 LI Y， LIN S， ZHANG B，et al ．Exploiting kernel sparsity and entropy for interpretable CNN compression ［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：2800-2809.
28 LIN W S， WU H N， HUANG C T ．Accelerating convolutional neural networks using iterative two-pass decomposition［C］∥ Proceedings of the 6th International Conference on Learning Representations．Vancouver：OpenReview.net，2018：1-11.
29 HUANG J， SUN W， HUANG L，et al ．Deep compression with low rank and sparse integrated decomposition［C］∥ Proceedings of 2019 IEEE the 7th International Conference on Computer Science and Network Technology．Dalian：IEEE，2019：289-292.
30 ALDROUBI A， HAMM K， KOKU A B，et al ．CUR decompositions，similarity matrices，and subspace clustering［J］．Frontiers in Applied Mathematics and Statistics，2019，4：65/1-16.
Options
Outlines

/

〈 〉