计算机科学与技术

基于密集特征推理及混合损失函数的修复算法

展开
  • 1.云南大学 信息学院,云南 昆明 650500
    2.云南大学 云南大学学报(自然科学版)编辑部,云南 昆明 650500
李海燕(1976-),女,教授,博士生导师,主要从事人工智能、图像处理研究。

收稿日期: 2022-07-04

  网络出版日期: 2023-02-06

基金资助

国家自然科学基金资助项目(62266049);云南省万人计划“云岭教学名师”(2019010015)

Image Inpainting Algorithm Based on Dense Feature Reasoning and Mix Loss Function

Expand
  • 1.School of Information Science and Engineering,Yunnan University,Kunming 650500,Yunnan,China
    2.Editorial Department of Journal of Yunnan University(Natural Science Edition),Yunnan University,Kunming 650500,Yunnan,China
李海燕(1976-),女,教授,博士生导师,主要从事人工智能、图像处理研究。

Received date: 2022-07-04

  Online published: 2023-02-06

Supported by

the National Natural Science Foundation of China(62266049)

摘要

为有效解决现有算法修复大面积不规则缺失图像时存在特征利用率低、图像结构连贯性差的问题,提出基于密集特征推理(DFR)及混合损失函数的图像修复算法。修复网络由多个特征推理(FR)模块密集连接组成,首先将待修复图像输入第1个推理模块中进行特征推理,之后将输出特征图通道合并送入下一个推理模块,后续推理的每一个模块的输入都是来自前面所有推理模块的推理特征,如此循环,以充分利用每个推理模块捕获的特征信息;然后提出一个传播一致性注意力机制(PCA),提高修补区域与已知区域的整体一致性;最后,提出混合损失函数(ML)优化修复结果的结构连贯性。整个DFR网络使用组归一化(GN),小批量训练也可达到优异的修复效果。在国际公认的Paris StreetView巴黎街景数据集和CelebA人脸数据集上验证文中所提算法的性能,主客观的实验结果表明:所提算法能有效修复大面积不规则缺失图像,提升特征利用率与结构连贯性,其平均峰值信噪比(PSNR)、平均结构相似度(SSIM)、均方误差(MSE)、弗雷歇距离(FID)及学习感知图像块相似度(LPIPS)指标优于对比算法。

本文引用格式

李海燕, 尹浩林, 李鹏, 等 . 基于密集特征推理及混合损失函数的修复算法[J]. 华南理工大学学报(自然科学版), 2023 , 51(9) : 99 -109 . DOI: 10.12141/j.issn.1000-565X.220420

Abstract

To effectively solve the problems of low feature utilization and poor image structure coherence occurred when existing algorithms are used to repair large irregularly missing images, this study proposed an image repair algorithm based on dense feature inference (DFR) and hybrid loss function. The repair network consists of multiple inference modules (FRs) densely connected. Firstly, after the image to be restored was fed into the first inference module for feature inference, the output feature map channels were merged and sent to the next inference module. The input of each subsequent inference module was the inferred features from all the previous inference modules and so on, so as to make full use of the feature information captured by each reasoning module. Subsequently, a propagation consistent attention (PCA) mechanism was proposed to improve the overall consistency of the patched regions with the known regions. Finally, a hybrid loss function (ML) was proposed to optimize the structural coherence of the repair results. The whole DFR network adopted group normalization (GN), and excellent repair results can be achieved even using small training batches. The performance of the proposed algorithm was verified on Paris StreetView and CelebA face datasets, which are internationally recognized datasets. The objective and subjective experimental results show that the proposed algorithm can effectively repair large irregular missing images, improve feature utilization and structural coherence. Its average peak signal-to-noise ratio (PSNR), average structural similarity ( SSIM), mean square error (MSE), Fréchet distance (FID) and learning perceptual image block similarity (LPIPS) metrics all outperform the comparison algorithms.

参考文献

1 EFROS A, LEUNG T K .Texture synthesis by non-parametric sampling[C]∥Proceedings of the Seventh IEEE International Conference on Computer Vision.Kerkyra:IEEE,1999:1033-1038.
2 CRIMINISI A, PéREZ P, TOYAMA K .Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing200413(9):1200-1212.
3 BARNES C, SHECHTMAN E, FINKELSTEIN A,et al .PatchMatch:A randomized correspondence algorithm for structural image editing[J].ACM Transactions on Graphics200928(3):1-11.
4 HE K, SUN J .Statistics of patch offsets for image completion[C]∥Proceedings of the European Conference on Computer Vision.Heidelberg,Berlin:Springer,2012:16-29.
5 MAO X, SHEN C, YANG Y B .Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[J].Advances in Neural Information Processing Systems201629:2810-2818.
6 K?HLER R, SCHULER C, SCH?LKOPF B,et al .Mask-specific inpainting with deep neural networks[C]∥Proceedings of the German Conference on Pattern Recognition.Cham:Springer,2014:523-534.
7 PATHAK D, KRAHENBUHL P, DONAHUE J,et al .Context encoders:Feature learning by inpainting[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE,2016:2536-2544.
8 LI Y, LIU S, YANG J,et al .Generative face completion[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii:IEEE,2017:3911-3919.
9 李海燕,吴自莹,郭磊,等 .基于混合空洞卷积网络的多鉴别器图像修复[J].华中科技大学学报(自然科学版)202149(3):40-45.
  LI Haiyan, WU Ziying, GUO Lei,et al .Multi-discriminator image inpainting algorithm based on hybrid dilated convolution network[J].Journal of Huazhong University of Science and Technology (Natural Science Edition)202149(3):40-45.
10 ZHAO L, MO Q, LIN S,et al .Uctgan:Diverse image inpainting based on unsupervised cross-space translation[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:5741-5750.
11 CAO C, FU Y .Learning a sketch tensor space for image inpainting of man-made scenes[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal:IEEE,2021:14509-14518.
12 刘微容,米彦春,杨帆,等 .基于多级解码网络的图像修复[J].电子学报202250(3):625-636.
  LIU Weirong, MI Yanchun, YANG Fan,et al .Generative image inpainting with multi-stage decoding network[J].Acta Electronica Sinica202250(3):625-636.
13 LIU G, REDA F A, SHIH K J,et al .Image inpainting for irregular holes using partial convolutions[C]∥Proceedings of the European Conference on Computer Vision (ECCV).Munich:Springer,2018:85-100.
14 ZHENG C, CHAM T J, CAI J .Pluralistic image completion[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach:IEEE,2019:1438-1447.
15 LI J, HE F, ZHANG L,et al .Progressive reconstruction of visual structure for image inpainting[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision.Seoul:IEEE,2019:5962-5971.
16 GUO X, YANG H, HUANG D .Image inpainting via conditional texture and structure dual generation[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal:IEEE,2021:14134-14143.
17 LI J, WANG N, ZHANG L,et al .Recurrent feature reasoning for image inpainting[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:7760-7768.
18 WU Y, HE K .Group normalization[C]∥Proceedings of the European Conference on Computer Vision (ECCV).Munich:Springer,2018:3-19.
19 HUANG G, LIU Z, VAN D M L,et al .Densely connected convolutional networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii:IEEE,2017:4700-4708.
20 HE K, ZHANG X, REN S,et al .Deep residual learning for image recognition[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,IEEE,2016:770-778.
21 RONNEBERGER O, FISCHER P, BROX T .U-Net:Convolutional networks for biomedical image segmentation[C]∥Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham:Springer,2015:234-241.
22 ZHAO H, GALLO O, FROSIO I,et al .Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging20163(1):47-57.
文章导航

/