Journal of South China University of Technology(Natural Science) >

2023 , Vol. 51 >Issue 9: 99 - 109

DOI: https://doi.org/10.12141/j.issn.1000-565X.220420

Computer Science & Technology

Image Inpainting Algorithm Based on Dense Feature Reasoning and Mix Loss Function

Expand
  • 1.School of Information Science and Engineering，Yunnan University，Kunming 650500，Yunnan，China
    2.Editorial Department of Journal of Yunnan University（Natural Science Edition），Yunnan University，Kunming 650500，Yunnan，China
李海燕（1976-），女，教授，博士生导师，主要从事人工智能、图像处理研究。

Received date: 2022-07-04

  Online published: 2023-02-06

Supported by

the National Natural Science Foundation of China(62266049)

Fold

Abstract

To effectively solve the problems of low feature utilization and poor image structure coherence occurred when existing algorithms are used to repair large irregularly missing images, this study proposed an image repair algorithm based on dense feature inference (DFR) and hybrid loss function. The repair network consists of multiple inference modules (FRs) densely connected. Firstly, after the image to be restored was fed into the first inference module for feature inference, the output feature map channels were merged and sent to the next inference module. The input of each subsequent inference module was the inferred features from all the previous inference modules and so on, so as to make full use of the feature information captured by each reasoning module. Subsequently, a propagation consistent attention (PCA) mechanism was proposed to improve the overall consistency of the patched regions with the known regions. Finally, a hybrid loss function (ML) was proposed to optimize the structural coherence of the repair results. The whole DFR network adopted group normalization (GN), and excellent repair results can be achieved even using small training batches. The performance of the proposed algorithm was verified on Paris StreetView and CelebA face datasets, which are internationally recognized datasets. The objective and subjective experimental results show that the proposed algorithm can effectively repair large irregular missing images, improve feature utilization and structural coherence. Its average peak signal-to-noise ratio (PSNR), average structural similarity ( SSIM), mean square error (MSE), Fréchet distance (FID) and learning perceptual image block similarity (LPIPS) metrics all outperform the comparison algorithms.

Key words： image inpainting; dense feature reasoning; attention mechanism; hybrid loss function; group normalization

Cite this article

LI Haiyan, YIN Haolin, LI Peng, et al. . Image Inpainting Algorithm Based on Dense Feature Reasoning and Mix Loss Function[J]. Journal of South China University of Technology(Natural Science), 2023 , 51(9) : 99 -109 . DOI: 10.12141/j.issn.1000-565X.220420

References

1 EFROS A， LEUNG T K ．Texture synthesis by non-parametric sampling［C］∥Proceedings of the Seventh IEEE International Conference on Computer Vision．Kerkyra：IEEE，1999：1033-1038．
2 CRIMINISI A， PéREZ P， TOYAMA K ．Region filling and object removal by exemplar-based image inpainting［J］．IEEE Transactions on Image Processing，2004，13（9）：1200-1212．
3 BARNES C， SHECHTMAN E， FINKELSTEIN A，et al ．PatchMatch：A randomized correspondence algorithm for structural image editing［J］．ACM Transactions on Graphics，2009，28（3）：1-11．
4 HE K， SUN J ．Statistics of patch offsets for image completion［C］∥Proceedings of the European Conference on Computer Vision．Heidelberg，Berlin：Springer，2012：16-29．
5 MAO X， SHEN C， YANG Y B ．Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections［J］．Advances in Neural Information Processing Systems，2016，29：2810-2818．
6 K?HLER R， SCHULER C， SCH?LKOPF B，et al ．Mask-specific inpainting with deep neural networks［C］∥Proceedings of the German Conference on Pattern Recognition．Cham：Springer，2014：523-534．
7 PATHAK D， KRAHENBUHL P， DONAHUE J，et al ．Context encoders：Feature learning by inpainting［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：2536-2544．
8 LI Y， LIU S， YANG J，et al ．Generative face completion［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Honolulu，Hawaii：IEEE，2017：3911-3919．
9 李海燕，吴自莹，郭磊，等 ．基于混合空洞卷积网络的多鉴别器图像修复［J］．华中科技大学学报（自然科学版），2021，49（3）：40-45．
  LI Haiyan， WU Ziying， GUO Lei，et al ．Multi-discriminator image inpainting algorithm based on hybrid dilated convolution network［J］．Journal of Huazhong University of Science and Technology （Natural Science Edition），2021，49（3）：40-45．
10 ZHAO L， MO Q， LIN S，et al ．Uctgan：Diverse image inpainting based on unsupervised cross-space translation［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：5741-5750．
11 CAO C， FU Y ．Learning a sketch tensor space for image inpainting of man-made scenes［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：14509-14518．
12 刘微容，米彦春，杨帆，等 ．基于多级解码网络的图像修复［J］．电子学报，2022，50（3）：625-636．
  LIU Weirong， MI Yanchun， YANG Fan，et al ．Generative image inpainting with multi-stage decoding network［J］．Acta Electronica Sinica，2022，50（3）：625-636．
13 LIU G， REDA F A， SHIH K J，et al ．Image inpainting for irregular holes using partial convolutions［C］∥Proceedings of the European Conference on Computer Vision （ECCV）．Munich：Springer，2018：85-100．
14 ZHENG C， CHAM T J， CAI J ．Pluralistic image completion［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：1438-1447．
15 LI J， HE F， ZHANG L，et al ．Progressive reconstruction of visual structure for image inpainting［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Seoul：IEEE，2019：5962-5971．
16 GUO X， YANG H， HUANG D ．Image inpainting via conditional texture and structure dual generation［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：14134-14143．
17 LI J， WANG N， ZHANG L，et al ．Recurrent feature reasoning for image inpainting［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：7760-7768．
18 WU Y， HE K ．Group normalization［C］∥Proceedings of the European Conference on Computer Vision （ECCV）．Munich：Springer，2018：3-19．
19 HUANG G， LIU Z， VAN D M L，et al ．Densely connected convolutional networks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Honolulu，Hawaii：IEEE，2017：4700-4708．
20 HE K， ZHANG X， REN S，et al ．Deep residual learning for image recognition［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas，IEEE，2016：770-778．
21 RONNEBERGER O， FISCHER P， BROX T ．U-Net：Convolutional networks for biomedical image segmentation［C］∥Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention．Cham：Springer，2015：234-241．
22 ZHAO H， GALLO O， FROSIO I，et al ．Loss functions for image restoration with neural networks［J］．IEEE Transactions on Computational Imaging，2016，3（1）：47-57．
Options
Outlines

/

〈 〉