用于视频压缩感知的特征域优化启发及多假设交叉注意力重构神经网络

doi:10.12141/j.issn.1000-565X.230578

摘要/Abstract

摘要：

现有视频压缩感知重建网络通常利用光流网络实现像素域运动估计与运动补偿。然而在重建过程中，光流网络的输入为质量较差的初始估计帧，导致获得的光流不准确，基于光流的像素域对齐与融合操作会造成噪声的累积，导致视频重建帧存在明显的人工效应，影响重建质量。基于特征域多通道信息对干扰噪声具有较强的鲁棒性，文中将特征域优化思想应用于视频压缩感知重构神经网络的设计中，提出了特征域优化启发及光流引导的多假设交叉注意力重构神经网络（FOFMCNet）。为避免光流中的噪声在图像变形时破坏图像结构的问题，文中在特征域设计了光流指导的多假设运动估计模块与基于交叉注意力的运动补偿模块，以实现特征域的帧间运动估计与运动补偿，从而更为充分地利用帧间相关性辅助非关键帧重构。为了在特征优化过程中加强有效信息的复用，提升网络学习能力并缓解梯度爆炸问题，文中设计了特征域优化启发U型网络（FOUNet），并作为FOFMCNet的子网络，通过多个FOUNet的级联，FOFMCNet在特征域实现非关键帧的优化与重建。实验结果表明，文中所提算法在经典低分辨率数据集（UCF-101和QCIF）和新的高分辨率数据集（REDS4）上的重构结果均优于现有的视频压缩感知算法。

关键词: 视频压缩感知, 特征域优化, 卷积神经网络, 注意力机制, 运动估计与补偿

Abstract:

The existing video compressive sensing reconstruction network usually uses the optical flow network to achieve pixel domain motion estimation and motion compensation. However, during the reconstruction process, the input of the optical flow network is the estimated frame with poor quality, resulting in inaccurate optical flow. The optical flow-based pixel domain alignment and fusion operation will cause noise accumulation, lead to obvious artificial effects in video reconstruction frames and affect the reconstruction quality. Based on the fact that multi-channel information in the feature space has strong robustness to interference noise, this paper applied the idea of feature space optimization to the design of the video compressive sensing reconstruction neural network, and proposed a feature-space optimization-inspired and flow-guided multi-hypothesis cross-attention network (FOFMCNet). To avoid the image structure destruction caused by the noise in the optical flow when warping the image, the study designed multi-hypothesis motion estimation module guided by optical flow and the motion compensation module based on cross-attention to realize the motion estimation and motion compensation of inter-frame in feature space, so as to make full use of inter-frame correlation to assist non-key frame reconstruction. In order to strengthen the reuse of effective information in the process of feature optimization, improve the learning ability of the network and alleviate the problem of gradient explosion, this paper designed a feature-space optimization-inspired u-shape network (FOUNet) as a sub-network of FOFMCNet. Through the cascade of multiple FOUNets, the FOFMCNet realizes the optimization and reconstruction of non-key frames in the feature space. Experimental results show that the reconstruction results of the proposed algorithm are obviously better than those of the existing video compression sensing algorithms on the classical low-resolution dataset (UCF-101 and QCIF) and new high-resolution dataset (REDS4).

Key words: video compressive sensing, feature-space optimization, convolutional neural network, attention mechanism, motion estimation and compensation

中图分类号:

TN919.81

杨春玲, 陈文俊, 刘嘉惠. 用于视频压缩感知的特征域优化启发及多假设交叉注意力重构神经网络[J]. 华南理工大学学报(自然科学版), 2024, 52(10): 9-21.

YANG Chunling, CHEN Wenjun, LIU Jiahui. Feature-Space Optimization-Inspired and Multi-Hypothesis Cross-Attention Reconstruction Neural Network for Video Compressive Sensing[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 9-21.

图/表 13

图1

图2

图3

图4

图5

图6

表1

表2

表3

图7

表4

表5

表6

参考文献 32

1	DONOHO D L ．Compressed sensing［J］．IEEE Tran-sactions on Information Theory，2006，52（4）：1289-1306.
2	MIDDYA R， CHAKRAVARTY N， NASKAR M K ．Compressive sensing in wireless sensor networks：a survey［J］．IETE Technical Review，2017，34（6）：642-654.
3	ZHOU S， XU T， DONG S，et al ．RDFNet：regional dynamic FISTA-Net for spectral snapshot compressive imaging［J］．IEEE Transactions on Computational Imaging，2023，9：490-501.
4	杨婧，苗延巍．压缩感知技术在颅脑MRI的临床应用进展［J］．磁共振成像，2022，13（9）：144-147，159.
	YANG Jing， MIAO Yanwei ．Clinical application progress of compressed sensing technology in cranial MRI［J］．Magnetic Resonance Imaging，2022，13（9）：144-147，159.
5	WANG Z， ZHANG H， CHENG Z，et al ．MetaSCI：scalable and adaptive reconstruction for video compressive sensing［C］∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021： 2083-2092.
6	BARON D， DUARTE M F， WAKIN M B，et al ．Distributed compressive sensing［EB/OL］．（2009-01-22）［2023-10-08］．.
7	韩乐，江怡华．鲁棒截断L₁-L₂全变分稀疏恢复模型［J］．华南理工大学学报（自然科学版），2023，51（5）：45-53.
	HAN Le， JIANG Yihua ．Robust truncated L₁-L₂ total variation sparse restoration models［J］．Journal of South China University of Technology （Natural Science Edition），2023，51（5）：45-53.
8	LIU Y， YUAN X， SUO J L，et al ．Rank minimization for snapshot compressive imaging［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，41（12）：2990-3006.
9	CHENG Z H， LU R Y， WANG Z J，et al ．BIRNAT：bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging［C］∥ Proceedings of the 16th European Conference on Computer Vision．Glasgow：Springer，2020：258-275.
10	CHENG Z H， CHEN B， LIU G L，et al ．Memory-efficient network for large-scale video compressive sensing［C］∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：16246-16255.
11	MUN S， FOWLER J E ．Residual reconstruction for block-based compressed sensing of video［C］∥ Proceedings of 2011 Data Compression Conference．Snowbird：IEEE，2011：183-192.
12	TRAMEL E W， FOWLER J E ．Video compressed sensing with multi hypothesis［C］∥ Proceedings of 2011 Data Compression Conference．Snowbird：IEEE，2011：193-202.
13	OU W F， YANG C L， LI W H，et al ．A two-stage multi-hypothesis reconstruction scheme in compressed video sensing［C］∥ Proceedings of 2016 IEEE International Conference on Image Processing．Phoenix：IEEE，2016：2494-2498.
14	ZHAO C， MA S， ZHANG J，et al ．Video compressive sensing reconstruction via reweighted residual sparsity［J］．IEEE Transactions on Circuits and Systems for Video Technology，2017，27（6）：1182-1195.
15	和志杰，杨春玲，汤瑞东．视频压缩感知中基于结构相似的帧间组稀疏表示重构算法研究［J］．电子学报，2018，46（3）：544-553.
	HE Zhi-jie， YANG Chun-ling， TANG Rui-dong ．Research on structural similarity based inter-frame group sparse representation for compressed video sensing［J］．Acta Electronica Sinica，2018，46（3）：544-553.
16	XU K， REN F ．CSVideoNet：a real-time end-to-end learning framework for high-frame-rate video compressive sensing［C］∥ Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision．Lake Tahoe：IEEE，2018：1680-1688.
17	LING X， YANG C， PEI H.Compressed video sensing network based on alignment prediction and residual reconstruction［C］∥ Proceedings of 2020 IEEE International Conference on Multimedia and Expo．London：IEEE，2020：9102723/1-6.
18	DAI J， QI H， XIONG Y，et al ．Deformable convolutional networks［C］∥ Proceedings of 2017 IEEE International Conference on Computer Vision.Venice：IEEE，2017：764-773.
19	魏志超，杨春玲．时域注意力特征对齐的视频压缩感知重构网络［J］．电子学报，2022，50（11）：2584-2592.
	WEI Zhi-chao， YANG Chun-ling ．Video compressed sensing reconstruction network based on temporal-attention feature alignment［J］．Acta Electronica Sinica，2022，50（11）：2584-2592.
20	XUAN Y， YANG C ．2sER-VGSR-Net：a two-stage enhancement reconstruction based on video group sparse representation network for compressed video sensing［C］∥ Proceedings of 2020 IEEE International Conference on Multimedia and Expo．London：IEEE，2020：9102849/1-6.
21	YANG X， YANG C ．ImrNet：an iterative motion compensation and residual reconstruction network for video compressed sensing［C］∥ Proceedings of 2021 IEEE International Conference on Acoustics，Speech and Signal Processing．Toronto：IEEE，2021：2350-2354.
22	RANJAN A， BLACK M J ．Optical flow estimation using a spatial pyramid network［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：4161-4170.
23	RONNEBERGER O， FISCHER P， BROX T ．U-Net：convolutional networks for biomedical image segmentation［C］∥ Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention．Munich：Springer，2015：234-241.
24	WEI Z， YANG C， XUAN Y ．Efficient video compressed sensing reconstruction via exploiting spatial-temporal correlation with measurement constraint［C］∥ Proceedings of 2021 IEEE International Conference on Multimedia and Expo．Shenzhen：IEEE，2021：9428203/1-6.
25	SUN D， YANG X， LIU M Y，et al ．PWC-Net：CNNs for optical flow using pyramid，warping，and cost volume［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：8934-8943.
26	CHEN W， YANG C， YANG X ．FSOINeT：feature-space optimization-inspired network for image compressive sensing［C］∥ Proceedings of 2022 IEEE International Conference on Acoustics，Speech and Signal Processing．Singapore：IEEE，2022：2460-2464.
27	MEHRI A， ARDAKANI P B， SAPPA A D ．MPRNet：multi-path residual network for lightweight image super resolution［C］∥ Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision．Waikoloa：IEEE，2021：2704-2713.
28	GAO Y， ZHANG M， WANG J，et al ．Cross-scale mixing attention for multisource remote sensing data fusion and classification［J］．IEEE Transactions on Geoscience and Remote Sensing，2023，61：5507815/1-15.
29	SOOMRO K， ZAMIR A R， SHAH M ．UCF101：a dataset of 101 human actions classes from videos in the wild［EB/OL］．（2012-12-03）［2023-10-08］．.
30	LOSHCHILOV I， HUTTER F ．SGDR：stochastic gradient descent with warm restarts［EB/OL］．（2017-05-03）［2023-10-08］．.
31	SHI W Z， LIU S H， JIANG F，et al ．Video compressed sensing using a convolutional neural network［J］．IEEE Transactions on Circuits and Systems for Video Technology，2020，31（2）：425-438.
32	YANG X， YANG C ．MAP-inspired deep unfolding network for distributed compressive video sensing［J］．IEEE Signal Processing Letters，2023，30：309-313.

算法	PSNR/dB			SSIM
算法	r_non-key=0.009	r_non-key=0.018	r_non-key=0.037	r_non-key=0.009	r_non-key=0.018	r_non-key=0.037
CSVideoNet	24.23	25.09	26.87	0.74	0.77	0.81
PRCVSNet	26.86	28.93	31.09
2sER-VGSR-Net	28.39	29.60	31.23	0.82	0.85	0.89
STMNet	29.98	31.14	32.50	0.89	0.91	0.93
JDR-TAFA-Net	30.33	31.63	33.14	0.89	0.91	0.94
ImrNet	30.54	31.90	33.40	0.89	0.91	0.93
FOFMCNet	30.98	32.68	34.17	0.90	0.93	0.95

序列	算法	PSNR/dB			SSIM
序列	算法	r_non-key=0.010	r_non-key=0.050	r_non-key=0.100	r_non-key=0.010	r_non-key=0.050	r_non-key=0.100
Hall	VCSNet	30.57	31.90	32.05	0.95	0.96	0.96
	PRCVSNet		33.81	35.61		0.97	0.98
	ImrNet	34.42	36.89	38.11	0.97	0.98	0.98
	JDR-TAFA-Net	35.39	37.26	38.25	0.98	0.98	0.98
	FOFMCNet	37.46	39.99	41.04	0.98	0.99	0.99
Ice	VCSNet	25.77	29.51	30.95	0.84	0.91	0.94
	PRCVSNet		31.67	33.66		0.95	0.97
	ImrNet	29.25	33.76	35.90	0.93	0.97	0.98
	JDR-TAFA-Net	30.03	34.83	36.23	0.94	0.97	0.98
	FOFMCNet	31.93	37.35	39.41	0.95	0.98	0.99
Soccer	VCSNet	24.62	28.62	30.51	0.63	0.77	0.85
	PRCVSNet		30.77	33.14		0.84	0.90
	ImrNet	27.55	31.81	34.24	0.92	0.88	0.92
	JDR-TAFA-Net	27.56	32.25	34.79	0.78	0.90	0.93
	FOFMCNet	28.84	34.35	36.87	0.81	0.93	0.95

r_non-key	算法	PSNR/dB					SSIM
r_non-key	算法	000序列	011序列	015序列	020序列	平均值	000序列	011序列	015序列	020序列	平均值
0.010	MH	22.48	18.79	23.89	17.47	20.66	0.65	0.47	0.72	0.44	0.57
	RRS	20.72	21.64	25.34	21.45	22.29	0.58	0.56	0.73	0.59	0.62
	VCSNet	24.28	23.95	28.10	23.82	25.03	0.64	0.58	0.78	0.59	0.65
	ImrNet	25.71	25.93	30.01	25.15	26.70	0.67	0.66	0.81	0.66	0.70
	DUMHAN	27.74	26.72	31.02	25.97	27.86	0.77	0.70	0.85	0.70	0.76
	FOFMCNet	29.12	27.57	31.59	26.71	28.75	0.84	0.73	0.86	0.74	0.79
0.100	MH	24.29	20.33	25.15	19.03	22.20	0.72	0.53	0.75	0.49	0.62
	RRS	26.95	28.92	34.33	27.90	29.53	0.80	0.80	0.92	0.80	0.83
	VCSNet	28.51	29.56	33.71	29.89	30.42	0.82	0.82	0.91	0.85	0.85
	ImrNet	29.09	32.29	36.33	31.23	32.24	0.83	0.88	0.94	0.89	0.88
	DUMHAN	31.80	33.52	38.00	32.17	33.87	0.91	0.90	0.95	0.91	0.92
	FOFMCNet	32.12	33.41	37.36	32.30	33.80	0.92	0.90	0.95	0.92	0.92

算法	参数量/10⁶	运行时间/s
ImrNet	8.13	0.622 1
DUMHAN	19.46	0.765 5
FOFMCNet	4.38	4.490 6

模块	PSNR/dB	SSIM
无FAAA	29.85	0.847 6
无光流	32.48	0.898 2
FOFMCNet	36.87	0.909 1