A TT-Tucker Decomposition-Based LC Convolutional Neural Network Compression Method Without Pre-Training

doi:10.12141/j.issn.1000-565X.230617

Abstract

Abstract:

Tensor training (TT) decomposition and Tucker decomposition are two effective compression methods for convolutional neural networks. However, TT and Tucker decomposition face the problems of spatial structure information loss and high computational complexity respectively. To solve the above problems, this paper considered the information retention rate and resource occupancy of the network structure and proposed a LC convolutional neural network compressed method (TT-LC) without pre-training based on TT-Tucker decomposition, adopting the learning-compression (LC) algorithm constraint compression framework. The TT-LC method includes two parts: learning step and compression step. The learning step didn’t not need the pre-training process, and adopted the exponential cyclic learning rate method to improve the training accuracy. In the compression step, this paper selected the global optimal rank according to the advantages of TT and Tucker decomposition and the characteristics of Bayes rule, and used empirically variable Bayesian matrix factorization (EVBMF) and Bayesian optimization (BayesOpt) to select reasonable ranks to guide tensor decomposition. The TT-LC method was used to compress the trained model. TT-LC method not only reduces the loss rate of spatial structure information and computational complexity, but also solves the problem that the unreasonable rank selection of the tensor leads to the significant decrease in model accuracy. It can realize the double Bayesian rank selection and double compression of the model, and obtains the optimal compression model. Finally, experiments were carried out on CIFAR10 and CIFAR100 datasets using ResNets and VGG networks. The results show that for ResNet32 network, compared with the benchmark method, the proposed method achieved a compression rate of parameter quantity of 69.6% and a floating point computation compression rate of 66.7% with the accuracy of 92.22%.

Key words: convolutional neural network, network compression, tensor decomposition, Bayesian optimization, constrained compression

CLC Number:

TP391.4

LIU Weirong, ZHANG Zhiqiang, ZHANG Ning, MENG Jiahao, ZHANG Min, LIU Jie. A TT-Tucker Decomposition-Based LC Convolutional Neural Network Compression Method Without Pre-Training[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(7): 29-38.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Experimental results of several methods with ResNet32 network on CIFAR10 dataset"

方法	$ε$	C/10⁶	γ_c/%	F/10⁶	γ_f /%	A/%
无压缩		0.46	0.0	68.86	0.0	92.49
LC（低压缩）	1.0×10^-4	0.25	45.7	36.09	47.6	92.21
PSTRN-S		0.18	60.9			91.44
LC（高压缩）	1.5×10^-4	0.17	63.0	27.01	60.8	91.62
LC-LR	4.0×10^-4	0.15	67.4	23.26	66.2	91.46
CUR		0.18	60.9	37.84	45.0	90.64
TT-LC	5.0×10^-6	0.14	69.6	22.93	66.7	92.22

Table 1

Table 2

Experimental results of several methods with ResNet56 network on CIFAR10 dataset"

方法	$ε$	C/10⁶	γ_c/%	F/10⁶	γ_f /%	A/%
无压缩		0.85	0.0	125.49	0.0	93.03
PF-B		0.73	14.1	90.90	27.6	93.06
NISP		0.36	57.6	70.76	43.6	93.01
AMC				62.75	50.0	91.90
ENC				62.75	50.0	93.00
CP				62.75	50.0	91.80
TRP		0.35	58.8			92.63
KSE		0.36	57.6	50.20	60.0	92.88
LC	1.0×10^-4	0.38	55.3	45.60	63.7	91.80
LC-LR	4.0×10^-4	0.28	67.1	41.83	66.7	92.54
CUR		0.37	56.5	80.44	35.9	91.58
TT-LC	5.0×10^-6	0.26	69.4	41.73	66.7	93.06

Table 2

Table 3

Experimental results of several methods with VGG16 network on CIFAR10 dataset"

方法	$ε$	C/10⁶	γ_c/%	F/10⁶	γ_f /%	A/%
无压缩		15.0	0.0	313.46	0.0	91.65
LSD		4.2	72.0			90.68
CPD		7.4	50.7			88.99
TT-LC	8.0×10^-5	6.0	60.0	127.71	59.3	91.54

Table 3

Table 4

Table 5

Fig.6

References 30

1	PUNYANI P， GUPTA R， KUMAR A ．Neural networks for facial age estimation：a survey on recent advances［J］．Artificial Intelligence Review，2020，53（5）：3299-3347.
2	SU N， CHEN X， GUAN J，et al ．Maritime target detection based on radar graph data and graph convolutional network［J］．IEEE Geoscience and Remote Sensing Letters，2022，19：4019705/1-5.
3	CONG S， ZHOU Y ．A review of convolutional neural network architectures and their optimizations［J］．Artificial Intelligence Review，2022，56（3）：1905-1969.
4	高晗，田育龙，许封元，等．深度学习模型压缩与加速综述［J］．软件学报，2021，32（1）：68-92.
	GAO Han， TIAN Yu-long， XU Feng-yuan，et al ．Survey of deep learning model compression and acceleration［J］．Journal of Software，2021，32（1）：68-92.
5	IDELBAYEV Y， CARREIRA-PERPIÑÁN M Á ．Low-rank compression of neural nets：learning the rank of each layer［C］∥ Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：8046-8056.
6	魏钰轩，陈莹．基于自适应层信息熵的卷积神经网络压缩［J］．电子学报，2022，50（10）：2398-2408.
	WEI Yu-xuan， CHEN Ying ．Convolutional neural network compression based on adaptive layer entropy［J］．Acta Electronica Sinica，2022，50（10）：2398-2408.
7	WU J， CONG L， WANG Y，et al ．Quantized convolutional neural networks for mobile devices［C］∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2016：4820-4828.
8	JI M， SHIN S， HWANG S，et al ．Refine myself by teaching myself：feature refinement via self-knowledge distillation［C］∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：10659-10668.
9	ELSKEN T， METZEN J H， HUTTER F ．Neural architecture search：a survey［J］．Journal of Machine Learning Research，2019，20：1997-2017.
10	LEBEDEV V， GANIN Y， RAKHUBA M，et al ．Speeding-up convolutional neural networks using fine-tuned CP-decomposition［C］∥ Proceedings of the 3rd International Conference on Learning Representations.San Diego：OpenReview.net，2015：1-11.
11	NOVIKOV A， PODOPRIKHIN D， OSOKIN A，et al ．Tensorizing neural networks［C］∥ Proceedings of the 28th International Conference on Neural Information Processing Systems．Montreal：ACM，2015：442-450.
12	WANG W， SUN Y， ERIKSSON B，et al ．Wide compression：tensor ring nets［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：9329-9338.
13	KIM Y D， PARK E， YOO S，et al ．Compression of deep convolutional neural networks for fast and low power mobile applications［C］∥ Proceedings of the 4th International Conference on Learning Representations.San Juan：OpenReview.net，2016：576-584.
14	NAKAJIMA S， SUGIYAMA M， BABACAN S D，et al ．Global analytic solution of fully-observed variational Bayesian matrix factorization［J］．Journal of Machine Learning Research，2013，14（1）：1-37.
15	KIM T， LEE J， CHOE Y ．Bayesian optimization-based global optimal rank selection for compression of convolutional neural networks［J］．IEEE Access，2020，8：17605-17618.
16	KOLDA T G， BADER B W ．Tensor decompositions and applications［J］．SIAM Review，2009，51（3）：455-500.
17	CHENG Z， LI B， FAN Y，et al ．A novel rank selection scheme in tensor ring decomposition based on reinforcement learning for deep neural networks［C］∥ Proceedings of 2020 IEEE International Conference on Acoustics，Speech and Signal Processing．Barcelona：IEEE，2020：3292-3296.
18	LI N， PAN Y， CHEN Y，et al ．Heuristic rank selection with progressively searching tensor ring network［J］．Complex & Intelligent Systems，2022，8：771-785.
19	BESAG J ．On the statistical-analysis of dirty pictures［J］．Journal of the Royal Statistical Society Series B：Methodological，1986，48（3）：259-302.
20	CAI G Y， LI J H， LIU X X，et al ．Learning and compressing：low-rank matrix factorization for deep neural network compression［J］．Applied Sciences，2023，13：2704/1-22.
21	XU Y， LI Y， ZHANG S，et al ．Traned rank pruning for efficient deep neural networks［C］∥ Proceedings of 2019 the Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition．Vancouver：IEEE，2019：14-17.
22	HE Y， ZHANG X， SUN J ．Channel pruning for accele-rating very deep neural networks［C］∥ Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：1398-1406.
23	HE Y， LIN J， LIU Z，et al ．AMC：autoML for model compression and acceleration on mobile devices［C］∥ Proceedings of the 15th European Conference on Computer Vision．Munich：Springer，2018：815-832.
24	LI H， KADAV A， DURDANOVIC I，et al ．Pruning filters for efficient ConvNets［C］∥ Proceedings of the 5th International Conference on Learning Representations．Toulon：OpenReview.net，2017：1-13.
25	YU R， LI A， CHEN C F，et al ．NISP：pruning networks using neuron importance score propagation［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：9194-9203.
26	KIM H， KHAN M U K， C-M KYUNG ．Efficient neural network compression［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：12561-12569.
27	LI Y， LIN S， ZHANG B，et al ．Exploiting kernel sparsity and entropy for interpretable CNN compression ［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：2800-2809.
28	LIN W S， WU H N， HUANG C T ．Accelerating convolutional neural networks using iterative two-pass decomposition［C］∥ Proceedings of the 6th International Conference on Learning Representations．Vancouver：OpenReview.net，2018：1-11.
29	HUANG J， SUN W， HUANG L，et al ．Deep compression with low rank and sparse integrated decomposition［C］∥ Proceedings of 2019 IEEE the 7th International Conference on Computer Science and Network Technology．Dalian：IEEE，2019：289-292.
30	ALDROUBI A， HAMM K， KOKU A B，et al ．CUR decompositions，similarity matrices，and subspace clustering［J］．Frontiers in Applied Mathematics and Statistics，2019，4：65/1-16.

方法	A/%	C/10⁶	γ_c/%	F/10⁶	γ_f /%
无压缩	91.25	0.27	0.0	40.6	0.0
LC-Tucker（高压缩率）	89.91	0.08	70.4	13.8	66.0
LC-Tucker（低压缩率）	90.23	0.09	66.7	14.4	64.5
TT-LC（no-BayesOpt）	90.32	0.09	66.7	14.7	63.8
TT-LC（固定秩为22）	89.08	0.08	70.4	15.1	62.8
TT-LC	90.37	0.08	70.4	13.1	67.7

[1]	DU Qiliang, WANG Yimin, TIAN Lianfang. Attention Module Based on Feature Similarity and Feature Normalization [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(7): 62-71.
[2]	MA Xiaoliang, AN Lingling, DENG Congjian, et al. Translation Optimization Technology of Automatic Speech Recognition Based on Industry-Specific Vocabulary [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(8): 118-125.
[3]	ZHU Zhengyu, LUO Chao, HE Qianhua, et al. Multi-View Lip Motion and Voice Consistency Judgment Based on Lip Reconstruction and Three-Dimensional Coupled CNN [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 70-77.
[4]	YE Feng, CHEN Biao, LAI Yizong. Contrastive Knowledge Distillation Method Based on Feature Space Embedding [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 13-23.
[5]	LUO Yutao, GAO Qiang. Traffic Sign Detection Based on Channel Attention and Feature Enhancement [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(12): 64-72.
[6]	QIU Zhibin, LU Zuwen, WANG Haixiang, et al. Recognition of Bird Sounds Related to Power Grid Faults Based on Mel Spectrogram and Convolutional Neural Network [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(2): 129-136.
[7]	ZHANG Xiangzhu, ZHANG Lijia, SONG Yifan, et al. Obstacle Avoidance Algorithm for Unmanned Aerial Vehicle Vision Based on Deep Learning [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(1): 101-108, 131.
[8]	HUANG Min QI Haitao JIANG Chunlin. Coupled Collaborative Filtering Model Based on Attention Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(7): 59-65.
[9]	JI Tianyao WANG Tingshao. Building Energy Consumption Prediction Based on Word Embedding and Convolutional Neural Network [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(6): 40-48.
[10]	Qi LIU Bin Yu. Pavement Crack Recognition Algorithm Based on Transposed CNN [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(12): 124-132.
[11]	LI Bo RAO Haobo. Salient Object Detection Based on Feature Enhancement in Complex Scene [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(11): 135-144.
[12]	DU Qiliang, HUANG Liguang, TIAN Lianfang, et al. Recognition of Passengers＇Abnormal Behavior on Escalator Based on Video Monitoring [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(8): 10-21.
[13]	CHEN Shanxiong, HAN Xu, LIN Xiaoyu, et al. MSER and CNN-Based Method for Character Detection in Ancient Yi Books [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(6): 123-133.
[14]	FAN Zizhu, WANG Song, ZHANG Hong, et al. W-Net-Based Segmentation for Ｒemote Sensing Satellite Image of High Resolution [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(12): 114-124.
[15]	LIU Jianguo, FENG Yunjian, JI Guo, et al. Improved Stereo Matching Algorithm Based on PSMNet [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(1): 60-69,83.