Image Inpainting Algorithm Based on Dense Feature Reasoning and Mix Loss Function

doi:10.12141/j.issn.1000-565X.220420

Abstract

Abstract:

To effectively solve the problems of low feature utilization and poor image structure coherence occurred when existing algorithms are used to repair large irregularly missing images, this study proposed an image repair algorithm based on dense feature inference (DFR) and hybrid loss function. The repair network consists of multiple inference modules (FRs) densely connected. Firstly, after the image to be restored was fed into the first inference module for feature inference, the output feature map channels were merged and sent to the next inference module. The input of each subsequent inference module was the inferred features from all the previous inference modules and so on, so as to make full use of the feature information captured by each reasoning module. Subsequently, a propagation consistent attention (PCA) mechanism was proposed to improve the overall consistency of the patched regions with the known regions. Finally, a hybrid loss function (ML) was proposed to optimize the structural coherence of the repair results. The whole DFR network adopted group normalization (GN), and excellent repair results can be achieved even using small training batches. The performance of the proposed algorithm was verified on Paris StreetView and CelebA face datasets, which are internationally recognized datasets. The objective and subjective experimental results show that the proposed algorithm can effectively repair large irregular missing images, improve feature utilization and structural coherence. Its average peak signal-to-noise ratio (PSNR), average structural similarity ( SSIM), mean square error (MSE), Fréchet distance (FID) and learning perceptual image block similarity (LPIPS) metrics all outperform the comparison algorithms.

Key words: image inpainting, dense feature reasoning, attention mechanism, hybrid loss function, group normalization

CLC Number:

TP391.41

LI Haiyan, YIN Haolin, LI Peng, et al.. Image Inpainting Algorithm Based on Dense Feature Reasoning and Mix Loss Function[J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 99-109.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Table 2

References 22

1	EFROS A， LEUNG T K ．Texture synthesis by non-parametric sampling［C］∥Proceedings of the Seventh IEEE International Conference on Computer Vision．Kerkyra：IEEE，1999：1033-1038．
2	CRIMINISI A， PÉREZ P， TOYAMA K ．Region filling and object removal by exemplar-based image inpainting［J］．IEEE Transactions on Image Processing，2004，13（9）：1200-1212．
3	BARNES C， SHECHTMAN E， FINKELSTEIN A，et al ．PatchMatch：A randomized correspondence algorithm for structural image editing［J］．ACM Transactions on Graphics，2009，28（3）：1-11．
4	HE K， SUN J ．Statistics of patch offsets for image completion［C］∥Proceedings of the European Conference on Computer Vision．Heidelberg，Berlin：Springer，2012：16-29．
5	MAO X， SHEN C， YANG Y B ．Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections［J］．Advances in Neural Information Processing Systems，2016，29：2810-2818．
6	KÖHLER R， SCHULER C， SCHÖLKOPF B，et al ．Mask-specific inpainting with deep neural networks［C］∥Proceedings of the German Conference on Pattern Recognition．Cham：Springer，2014：523-534．
7	PATHAK D， KRAHENBUHL P， DONAHUE J，et al ．Context encoders：Feature learning by inpainting［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：2536-2544．
8	LI Y， LIU S， YANG J，et al ．Generative face completion［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Honolulu，Hawaii：IEEE，2017：3911-3919．
9	李海燕，吴自莹，郭磊，等．基于混合空洞卷积网络的多鉴别器图像修复［J］．华中科技大学学报（自然科学版），2021，49（3）：40-45．
	LI Haiyan， WU Ziying， GUO Lei，et al ．Multi-discriminator image inpainting algorithm based on hybrid dilated convolution network［J］．Journal of Huazhong University of Science and Technology （Natural Science Edition），2021，49（3）：40-45．
10	ZHAO L， MO Q， LIN S，et al ．Uctgan：Diverse image inpainting based on unsupervised cross-space translation［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：5741-5750．
11	CAO C， FU Y ．Learning a sketch tensor space for image inpainting of man-made scenes［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：14509-14518．
12	刘微容，米彦春，杨帆，等．基于多级解码网络的图像修复［J］．电子学报，2022，50（3）：625-636．
	LIU Weirong， MI Yanchun， YANG Fan，et al ．Generative image inpainting with multi-stage decoding network［J］．Acta Electronica Sinica，2022，50（3）：625-636．
13	LIU G， REDA F A， SHIH K J，et al ．Image inpainting for irregular holes using partial convolutions［C］∥Proceedings of the European Conference on Computer Vision （ECCV）．Munich：Springer，2018：85-100．
14	ZHENG C， CHAM T J， CAI J ．Pluralistic image completion［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：1438-1447．
15	LI J， HE F， ZHANG L，et al ．Progressive reconstruction of visual structure for image inpainting［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Seoul：IEEE，2019：5962-5971．
16	GUO X， YANG H， HUANG D ．Image inpainting via conditional texture and structure dual generation［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：14134-14143．
17	LI J， WANG N， ZHANG L，et al ．Recurrent feature reasoning for image inpainting［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：7760-7768．
18	WU Y， HE K ．Group normalization［C］∥Proceedings of the European Conference on Computer Vision （ECCV）．Munich：Springer，2018：3-19．
19	HUANG G， LIU Z， VAN D M L，et al ．Densely connected convolutional networks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Honolulu，Hawaii：IEEE，2017：4700-4708．
20	HE K， ZHANG X， REN S，et al ．Deep residual learning for image recognition［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas，IEEE，2016：770-778．
21	RONNEBERGER O， FISCHER P， BROX T ．U-Net：Convolutional networks for biomedical image segmentation［C］∥Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention．Cham：Springer，2015：234-241．
22	ZHAO H， GALLO O， FROSIO I，et al ．Loss functions for image restoration with neural networks［J］．IEEE Transactions on Computational Imaging，2016，3（1）：47-57．

数据集	掩膜率	算法	PSNR/dB	SSIM	MSE	FID	LPIPS
CelebA	30%~40%	PIC	23.25	0.836 5	0.027 8	65.259 9	0.116 7
		PRVS	25.05	0.856 3	0.021 6	57.310 6	0.088 2
		RFR	24.92	0.852 8	0.022 1	47.530 4	0.077 4
		CTSDG	25.39	0.859 1	0.020 4	56.690 0	0.087 3
		本文所提	25.50	0.867 9	0.020 2	41.563 9	0.067 2
	40%~50%	PIC	22.06	0.822 8	0.032 3	70.283 3	0.146 5
		PRVS	23.74	0.832 4	0.025 9	66.792 4	0.112 3
		RFR	23.52	0.828 7	0.026 7	52.232 3	0.097 8
		CTSDG	23.67	0.831 6	0.025 7	67.283 6	0.115 7
		本文所提	24.08	0.844 7	0.024 4	49.450 2	0.086 4
	50%~60%	PIC	20.40	0.749 5	0.048 3	98.730 6	0.203 0
		PRVS	22.00	0.760 1	0.038 2	87.411 0	0.149 2
		RFR	22.14	0.763 4	0.037 9	63.751 2	0.125 9
		CTSDG	22.18	0.768 0	0.037 0	84.335 1	0.148 3
		本文所提	22.53	0.780 8	0.035 6	60.557 1	0.111 9
Paris Street View	30%~40%	PIC	21.89	0.762 3	0.037 9	69.887 9	0.198 6
		PRVS	24.48	0.809 6	0.025 7	54.362 4	0.148 4
		RFR	24.33	0.807 2	0.026 6	47.291 3	0.128 9
		CTSDG	24.59	0.816 3	0.025 0	57.168 7	0.143 7
		本文所提	24.80	0.821 1	0.024 8	42.174 2	0.116 9
	40%~50%	PIC	20.15	0.683 7	0.052 5	92.187 4	0.265 3
		PRVS	22.83	0.745 6	0.034 8	73.046 6	0.198 1
		RFR	22.67	0.742 8	0.035 9	60.992 0	0.170 1
		CTSDG	22.97	0.751 9	0.033 8	75.180 3	0.191 4
		本文所提	23.04	0.759 6	0.033 7	53.938 7	0.155 7
	50%~60%	PIC	20.46	0.661 9	0.052 2	92.376 3	0.276 0
		PRVS	22.59	0.720 2	0.037 8	79.934 8	0.213 7
		RFR	22.40	0.717 4	0.039 0	69.280 2	0.186 5
		CTSDG	22.71	0.726 7	0.036 9	86.129 3	0.206 9
		本文所提	22.87	0.738 3	0.036 5	63.094 2	0.168 7

模型	PSNR/dB	SSIM	MSE	FID	LPIPS
无密集连接	23.47	0.803 9	0.032 4	62.066 6	0.115 0
有密集连接	23.57	0.805 7	0.031 1	57.810 6	0.106 1
无注意力传播	24.40	0.847 7	0.024 6	40.282 6	0.079 1
有注意力传播	24.50	0.850 6	0.024 4	41.061 3	0.082 0
单一损失函数	24.09	0.808 5	0.028 9	55.088 8	0.101 4
混合损失函数	24.26	0.818 0	0.028 7	59.101 7	0.101 5
批量归一化	23.02	0.793 8	0.032 7	66.446 8	0.112 4
组归一化	23.53	0.805 7	0.031 1	58.324 0	0.104 2

[1]	LI Jiachun, LI Bowen, LIN Weiwei. AdfNet: An Adaptive Deep Forgery Detection Network Based on Diverse Features [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 82-89.
[2]	GUO Enqiang, FU Xinsha. Dropped Object Detection Method Based on Feature Similarity Learning [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(6): 30-41.
[3]	LIU Yupeng, ZHANG Lei. Cognitive Diagnosis Model Integrating Forgetting and Importance of Knowledge Points [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 54-62.
[4]	LU Lu, LAI Jinxiong. Smart Contract Vulnerability Detection Method Based on Capsule Network and Attention Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 36-44.
[5]	WO Yan, LIANG Jiyun, HAN Guoqiang. A cross-modal face retrieval method based on metric learning [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(6): 1-9.
[6]	YU Ying, HE Penghao, XU Chaoyue . Image Inpainting via Residual Attention Fusion and Gated Information Distillation [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(12): 49-59.
[7]	YANG Jinsheng, CHEN Hongpeng, GUAN Xin, et al. A Multi-Scale Lightweight Brain Glioma Image Segmentation Network [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(12): 132-141.
[8]	HUANG Min QI Haitao JIANG Chunlin. Coupled Collaborative Filtering Model Based on Attention Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(7): 59-65.
[9]	LIU Huiting, LI Yinjie, GUO Lingling, et al. Tightly Coupled Recommendation Algorithm Based on Heterogeneous Information Networks [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(7): 66-75.
[10]	HU Guanghua, WANG Ning, HE Wenliang, et al. Unsupervised Surface Defect Detection Method Based on Image Inpainting [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(7): 76-85,124.
[11]	ZHANG Ruifeng, BAI Jintong, GUAN Xin, et al. Music Source Separation Method Based on Unet Combining SE and BiSRU [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(11): 106-115,134.
[12]	IKA Novita Dewi, CAI Xiaoling, et al. Drug-Drug Interaction Extraction Model Combining Category Keywords with Attention Mechanism [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(1): 10-17.
[13]	LIU Huiting, JI Qiang, LIU Huimin, et al. Joint Deep Recommendation Model Based on Double-Layer Attention Mechanism [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(6): 97-105.
[14]	Wang Wei-ning Liu Jian-cong Xu Xiang-min Jiang Yi-zi Wang Li. Aesthetic Enhancement of Images Based on Photography Composition Guidelines [J]. Journal of South China University of Technology (Natural Science Edition), 2015, 43(5): 51-58.