Deep Feature Domain Motion Estimation and Multi-Layer Multi-Hypothesis Motion Compensation Net for Video Compression Codec

doi:10.12141/j.issn.1000-565X.220221

Abstract

Abstract:

Traditional video compression coding methods are widely used. In order to further improve the compression performance, research on deep learning-based video compression coding methods has received increasing attention. Existing deep learning video compression coding methods realize motion compensation based on optical flow, which will produce artifacts during the optical flow alignment process, reducing the accuracy of prediction. This paper proposed a motion estimation idea in the deep feature domain, and designed a corresponding neural network to extract motion information in the deep feature domain. On this basis, it proposed a multi-layer multi-hypothesis prediction motion compensation network. By using the multi-hypothesis prediction module in the deep feature domain, the shallow feature domain and the pixel domain, the accuracy of motion compensation was improved, thereby improving the overall rate-distortion performance. Simulation results show that the inter-frame prediction results of the algorithm in the paper mitigate artifacts and the visual effect is significantly better than optical flow alignment. At the same time, the proposed algorithm achieves better rate-distortion performance compared with traditional H.264 and H.265 methods and single-frame reference methods DVC and DVCpro based on deep learning. Compared with the DCVC method at the forefront of research, the algorithm reduces the coding time by approximately 26.8% while the rate distortion performance is similar. Taking the H.264 encoding result as the benchmark, under the condition of the same bit rate, the decoding quality was improved by 3.73 dB, 4.76 dB and 2.65 dB on HEVC test sequences ClassB, ClassD and ClassE. The simulation experiment results show that, when compressing and coding video sequences, the algorithm proposed in the paper can improve the accuracy of motion compensation prediction frames, reduce the prediction error, shortens the residual signal compression coding code stream and improve the overall rate distortion performance.

Key words: video compression, deep learning, motion estimation, multi-hypothesis prediction, codec network

CLC Number:

TN919.8

YANG Chunling, LÜ Zeyu . Deep Feature Domain Motion Estimation and Multi-Layer Multi-Hypothesis Motion Compensation Net for Video Compression Codec[J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(10): 51-61.

References 24

1	WIEGAND T， SULLIVAN G J， BJONTEGAARD G，et al ．Overview of the H.264/AVC video coding standard ［J］．IEEE Transactions on Circuits and Systems for Video Technology，2003，13（7）：560-576.
2	SULLIVAN G J， OHM J R， HAN W J，et al ．Overview of the high efficiency video coding （HEVC） standard ［J］．IEEE Transactions on Circuits and Systems for Video Technology，2012，22（12）：1649-1668.
3	LU G， OUYANG W， XU D，et al ．DVC：An end-to-end deep video compression framework ［C］ ∥ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：11006-11015.
4	LU G， ZHANG X， OUYANG W，et al ．An end-to-end learning framework for video compression ［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2021，43（10）：3292-3308.
5	YANG X， YANG C ．ImrNet：An iterative motion compensation and residual reconstruction network for video compressed sensing ［C］ ∥ Proceedings of ICASSP 2021-2021 IEEE International Conference on Acoustics，Speech and Signal Processing. Toronto：IEEE，2021：2350-2354.
6	WEI Z， YANG C， XUAN Y ．Efficient video compressed sensing reconstruction via exploiting spatial-temporal correlation with measurement constraint ［C］ ∥ Proceedings of 2021 IEEE International Conference on Multimedia and Expo．Shenzhen：IEEE，2021：1-6.
7	禤韵怡，杨春玲．基于帧间组稀疏的两阶段递归增强视频压缩感知重构网络［J］．电子学报，2021，49（3）：435-442.
	XUAN Yunyi， YANG Chunling ．Two-stage recursive enhancement reconstruction based on video inter-frame group sparse representation in compressed video sensing ［J］．Acta Electronica Sinica，2021，49（3）：435-442.
8	HU Z， CHEN Z， XU D，et al ．Improving deep video compression by resolution-adaptive flow coding ［C］ ∥Proceedings of European Conference on Computer Vision．Edinburgh ：Springer，2020：193-209.
9	LU G， CAI C， ZHANG X，et al ．Content adaptive and error propagation aware deep video compression ［C］ ∥ Proceedings of European Conference on Computer Vision. Edinburgh：Springer，2020：456-472.
10	LIN J， LIU D， LI H，et al ．M-LVC：Multiple frames prediction for learned video compression ［C］ ∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：3546-3554.
11	ALEXANDRE D， HANG H M， PENG W H，et al ．Deep video compression for interframe coding ［C］ ∥ Proceedings of 2021 IEEE International Conference on Image Processing. Anchorage：IEEE，2021：2124-2128.
12	YANG R， MENTZER F， VAN GOOL L，et al ．Learning for video compression with recurrent auto-encoder and recurrent probability model ［J］．IEEE Journal of Selected Topics in Signal Processing，2020，15（2）：388-401.
13	SHI X， CHEN Z， WANG H，et al ．Convolutional LSTM network：A machine learning approach for precipitation nowcasting ［C］ ∥ Proceedings of Advances in Neural Information Processing Systems．Montreal：MIT Press，2015：28-44.
14	LI J， LI B， LU Y ．Deep contextual video compression ［C］ ∥ Proceedings of Advances in Neural Information Processing Systems．［S.l.］：MIT Press，2021：18114-18125.
15	DAI J， QI H， XIONG Y，et al ．Deformable convolutional networks ［C］ ∥ Proceedings of the IEEE International Conference on Computer Vision．Venice：IEEE，2017：764-773.
16	WANG X， CHAN K C K， YU K，et al ．EDVR：Video restoration with enhanced deformable convolutional networks ［C］ ∥ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops．Long Beach：IEEE，2019：1954-1963.
17	HU Z， LU G， XU D ．FVC：A new framework towards deep video compression in feature space ［C］ ∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Virtual：IEEE，2021：1502-1511.
18	LING X， YANG C， PEI H ．Compressed video sensing network based on alignment prediction and residual reconstruction ［C］ ∥ Proceedings of 2020 IEEE International Conference on Multimedia and Expo．London：IEEE，2020：1-6.
19	MINNEN D， BALLÉ J， TODERICI G D ．Joint autoregressive and hierarchical priors for learned image compression ［C］ ∥ Proceedings of Advances in Neural Information Processing Systems．Montreal：MIT Press，2018：10771-10780.
20	BALLÉ J， MINNEN D， SINGH S，et al ．Variational image compression with a scale hyperprior ［C］ ∥ Proceedings of International Conference on Learning Representations．Vancouver：［s.n.］，2018：1-23.
21	CHENG Z， SUN H， TAKEUCHI M，et al ．Learned image compression with discretized gaussian mixture likelihoods and attention modules ［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle：IEEE，2020：7939-7948.
22	XUE T， CHEN B， WU J，et al ．Video enhancement with task-oriented flow ［J］．International Journal of Computer Vision，2019，127（8）：1106-1125.
23	MERCAT A， VIITANEN M， VANNE J ．UVG dataset：50/120fps 4K sequences for video codec analysis and development ［C］ ∥Proceedings of the 11th ACM Multimedia Systems Conference．Istanbul：Association for Computing Machinery，2020：297-302.
24	BJONTEGAARD G. Calculation of average PSNR differences between RD-curves：VCEG-M33 ［Z］．Austin：ITU-T，2001.

[1]	. Feature-domain Proximal High-Dimensional Gradient Descent Network for Image Compressed Sensing [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(3): 119-130.
[2]	ZHENG Juanyi, DONG Jiahao, ZHANG Qingjue, et al. Reconfigurable Intelligence Surface Channel Estimation Algorithm Based on RDN [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(3): 102-111.
[3]	. Research on Forest Fire Recognition Based on Improved EfficientNet-E Model Based on ECA Attention Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(2): 42-49.
[4]	LI Fang, GUO Weisen, ZHANG Ping, et al.. Prediction Technique for Remaining Useful Life of Bearing Based on Spatial-Temporal Dual Cell State [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 69-81.
[5]	SU Jindian, YU Shanshan, HONG Xiaobin. A Self-Supervised Pre-Training Method for Chinese Spelling Correction [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 90-98.
[6]	LI Jiachun, LI Bowen, LIN Weiwei. AdfNet: An Adaptive Deep Forgery Detection Network Based on Diverse Features [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 82-89.
[7]	GUO Enqiang, FU Xinsha. Dropped Object Detection Method Based on Feature Similarity Learning [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(6): 30-41.
[8]	ZHAO Jiandong, JIAO Lanxin, ZHAO Zhimin, et al. A Car-Following Model Driven by Combination of Theory and Data Considering Effects of Lane Change of Side Cars [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(6): 10-19.
[9]	YE Feng, CHEN Biao, LAI Yizong. Contrastive Knowledge Distillation Method Based on Feature Space Embedding [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 13-23.
[10]	ZHAO Rongchao, WU Baili, CHEN Zhuyun, WEN Kairu, ZHANG Shaohui, LI Weihua. Graph Neural Network for Fault Diagnosis with Multi-Scale Time-Spatial Information Fusion Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(12): 42-52.
[11]	HOU Liwei, WANG Hengsheng, ZOU Haoran. Deep Learning-Based Prediction of Contact Force in the Process of Shoveling Up Glass Subtrate [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(8): 71-81.
[12]	MO Jianwen, ZHU Yanqiao, YUAN Hua, et al. Incremental learning based on neuron regularization and resource releasing [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(6): 71-79,90.
[13]	LU Lu, ZHONG Wenyu, WU Xiaokun. Image tampering localization based on mutil-scale transformer [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(6): 10-18.
[14]	ZHANG Qin, HU Jiahui, REN Hailin. Intelligent Pushing Method and Experiment of Feeding Assistant Robot [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(6): 111-120.
[15]	YANG Chunling, LING Xi, LÜ Zeyu. Feature-domain Multi-Hypothesis Prediction Reconstruction Neural Network for Compressed Video Sensing [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(6): 80-90.