收稿日期: 2023-10-07
网络出版日期: 2024-02-02
基金资助
国家自然科学基金资助项目(62261032);甘肃省自然科学基金资助项目(22JR5RA272);甘肃省重点人才项目
A TT-Tucker Decomposition-Based LC Convolutional Neural Network Compression Method Without Pre-Training
Received date: 2023-10-07
Online published: 2024-02-02
Supported by
the National Natural Science Foundation of China(62261032);the Natural Science Foundation of Gansu Province(22JR5RA272);the Key Talent Project of Gansu Province
张量训练(TT)分解和Tucker分解是两种有效的卷积神经网络压缩方法。然而,TT和Tucker分解分别面临空间结构信息丢失与计算复杂度高等问题。为解决上述问题,文中考虑了网络结构的信息保留率和资源占用情况,采用学习-压缩(LC)算法的约束型压缩框架,提出了一种基于TT-Tucker分解的无预训练LC卷积神经网络压缩方法(TT-LC)。TT-LC方法包括学习步骤和压缩步骤两个部分。学习步骤不需要预训练过程,采用了指数循环学习率方法以提高训练准确率。而在压缩步骤,文中根据TT和Tucker分解的优点以及贝叶斯规则选取全局最优秩的特性,运用经验变分贝叶斯矩阵分解(EVBMF)和贝叶斯优化(BayesOpt)选出合理的秩以指导张量分解,采用TT-LC方法压缩训练后的模型。TT-LC方法既降低了空间结构信息丢失率和计算复杂度,又解决了张量的秩选取不合理导致模型准确率显著下降的问题,可实现模型的双重贝叶斯选秩和双重压缩,获得最优的压缩模型。最后,采用ResNets和VGG网络在CIFAR10与CIFAR100数据集上进行实验。结果表明:对于ResNet32网络,相比于基准方法,文中方法在准确率为92.22%的情况下,获得了69.6%的参数量压缩率和66.7%的浮点计算量压缩率。
刘微容 , 张志强 , 张宁 , 孟家豪 , 张敏 , 刘婕 . 基于TT-Tucker分解的无预训练LC卷积神经网络压缩方法[J]. 华南理工大学学报(自然科学版), 2024 , 52(7) : 29 -38 . DOI: 10.12141/j.issn.1000-565X.230617
Tensor training (TT) decomposition and Tucker decomposition are two effective compression methods for convolutional neural networks. However, TT and Tucker decomposition face the problems of spatial structure information loss and high computational complexity respectively. To solve the above problems, this paper considered the information retention rate and resource occupancy of the network structure and proposed a LC convolutional neural network compressed method (TT-LC) without pre-training based on TT-Tucker decomposition, adopting the learning-compression (LC) algorithm constraint compression framework. The TT-LC method includes two parts: learning step and compression step. The learning step didn’t not need the pre-training process, and adopted the exponential cyclic learning rate method to improve the training accuracy. In the compression step, this paper selected the global optimal rank according to the advantages of TT and Tucker decomposition and the characteristics of Bayes rule, and used empirically variable Bayesian matrix factorization (EVBMF) and Bayesian optimization (BayesOpt) to select reasonable ranks to guide tensor decomposition. The TT-LC method was used to compress the trained model. TT-LC method not only reduces the loss rate of spatial structure information and computational complexity, but also solves the problem that the unreasonable rank selection of the tensor leads to the significant decrease in model accuracy. It can realize the double Bayesian rank selection and double compression of the model, and obtains the optimal compression model. Finally, experiments were carried out on CIFAR10 and CIFAR100 datasets using ResNets and VGG networks. The results show that for ResNet32 network, compared with the benchmark method, the proposed method achieved a compression rate of parameter quantity of 69.6% and a floating point computation compression rate of 66.7% with the accuracy of 92.22%.
| 1 | PUNYANI P, GUPTA R, KUMAR A .Neural networks for facial age estimation:a survey on recent advances[J].Artificial Intelligence Review,2020,53(5):3299-3347. |
| 2 | SU N, CHEN X, GUAN J,et al .Maritime target detection based on radar graph data and graph convolutional network[J].IEEE Geoscience and Remote Sensing Letters,2022,19:4019705/1-5. |
| 3 | CONG S, ZHOU Y .A review of convolutional neural network architectures and their optimizations[J].Artificial Intelligence Review,2022,56(3):1905-1969. |
| 4 | 高晗,田育龙,许封元,等 .深度学习模型压缩与加速综述[J].软件学报,2021,32(1):68-92. |
| GAO Han, TIAN Yu-long, XU Feng-yuan,et al .Survey of deep learning model compression and acceleration[J].Journal of Software,2021,32(1):68-92. | |
| 5 | IDELBAYEV Y, CARREIRA-PERPI?áN M á .Low-rank compression of neural nets:learning the rank of each layer[C]∥ Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:8046-8056. |
| 6 | 魏钰轩,陈莹 .基于自适应层信息熵的卷积神经网络压缩[J].电子学报,2022,50(10):2398-2408. |
| WEI Yu-xuan, CHEN Ying .Convolutional neural network compression based on adaptive layer entropy[J].Acta Electronica Sinica,2022,50(10):2398-2408. | |
| 7 | WU J, CONG L, WANG Y,et al .Quantized convolutional neural networks for mobile devices[C]∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2016:4820-4828. |
| 8 | JI M, SHIN S, HWANG S,et al .Refine myself by teaching myself:feature refinement via self-knowledge distillation[C]∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Nashville:IEEE,2021:10659-10668. |
| 9 | ELSKEN T, METZEN J H, HUTTER F .Neural architecture search:a survey[J].Journal of Machine Learning Research,2019,20:1997-2017. |
| 10 | LEBEDEV V, GANIN Y, RAKHUBA M,et al .Speeding-up convolutional neural networks using fine-tuned CP-decomposition[C]∥ Proceedings of the 3rd International Conference on Learning Representations.San Diego:OpenReview.net,2015:1-11. |
| 11 | NOVIKOV A, PODOPRIKHIN D, OSOKIN A,et al .Tensorizing neural networks[C]∥ Proceedings of the 28th International Conference on Neural Information Processing Systems.Montreal:ACM,2015:442-450. |
| 12 | WANG W, SUN Y, ERIKSSON B,et al .Wide compression:tensor ring nets[C]∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE,2018:9329-9338. |
| 13 | KIM Y D, PARK E, YOO S,et al .Compression of deep convolutional neural networks for fast and low power mobile applications[C]∥ Proceedings of the 4th International Conference on Learning Representations.San Juan:OpenReview.net,2016:576-584. |
| 14 | NAKAJIMA S, SUGIYAMA M, BABACAN S D,et al .Global analytic solution of fully-observed variational Bayesian matrix factorization[J].Journal of Machine Learning Research,2013,14(1):1-37. |
| 15 | KIM T, LEE J, CHOE Y .Bayesian optimization-based global optimal rank selection for compression of convolutional neural networks[J].IEEE Access,2020,8:17605-17618. |
| 16 | KOLDA T G, BADER B W .Tensor decompositions and applications[J].SIAM Review,2009,51(3):455-500. |
| 17 | CHENG Z, LI B, FAN Y,et al .A novel rank selection scheme in tensor ring decomposition based on reinforcement learning for deep neural networks[C]∥ Proceedings of 2020 IEEE International Conference on Acoustics,Speech and Signal Processing.Barcelona:IEEE,2020:3292-3296. |
| 18 | LI N, PAN Y, CHEN Y,et al .Heuristic rank selection with progressively searching tensor ring network[J].Complex & Intelligent Systems,2022,8:771-785. |
| 19 | BESAG J .On the statistical-analysis of dirty pictures[J].Journal of the Royal Statistical Society Series B:Methodological,1986,48(3):259-302. |
| 20 | CAI G Y, LI J H, LIU X X,et al .Learning and compressing:low-rank matrix factorization for deep neural network compression[J].Applied Sciences,2023,13:2704/1-22. |
| 21 | XU Y, LI Y, ZHANG S,et al .Traned rank pruning for efficient deep neural networks[C]∥ Proceedings of 2019 the Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition.Vancouver:IEEE,2019:14-17. |
| 22 | HE Y, ZHANG X, SUN J .Channel pruning for accele-rating very deep neural networks[C]∥ Proceedings of 2017 IEEE International Conference on Computer Vision.Venice:IEEE,2017:1398-1406. |
| 23 | HE Y, LIN J, LIU Z,et al .AMC:autoML for model compression and acceleration on mobile devices[C]∥ Proceedings of the 15th European Conference on Computer Vision.Munich:Springer,2018:815-832. |
| 24 | LI H, KADAV A, DURDANOVIC I,et al .Pruning filters for efficient ConvNets[C]∥ Proceedings of the 5th International Conference on Learning Representations.Toulon:OpenReview.net,2017:1-13. |
| 25 | YU R, LI A, CHEN C F,et al .NISP:pruning networks using neuron importance score propagation[C]∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE,2018:9194-9203. |
| 26 | KIM H, KHAN M U K, C-M KYUNG .Efficient neural network compression[C]∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach:IEEE,2019:12561-12569. |
| 27 | LI Y, LIN S, ZHANG B,et al .Exploiting kernel sparsity and entropy for interpretable CNN compression [C]∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach:IEEE,2019:2800-2809. |
| 28 | LIN W S, WU H N, HUANG C T .Accelerating convolutional neural networks using iterative two-pass decomposition[C]∥ Proceedings of the 6th International Conference on Learning Representations.Vancouver:OpenReview.net,2018:1-11. |
| 29 | HUANG J, SUN W, HUANG L,et al .Deep compression with low rank and sparse integrated decomposition[C]∥ Proceedings of 2019 IEEE the 7th International Conference on Computer Science and Network Technology.Dalian:IEEE,2019:289-292. |
| 30 | ALDROUBI A, HAMM K, KOKU A B,et al .CUR decompositions,similarity matrices,and subspace clustering[J].Frontiers in Applied Mathematics and Statistics,2019,4:65/1-16. |
/
| 〈 |
|
〉 |