Journal of South China University of Technology(Natural Science) >

2025 , Vol. 53 >Issue 3: 31 - 39

DOI: https://doi.org/10.12141/j.issn.1000-565X.240155

Computer Science & Technology

Image Inpainting Algorithm Based on Hybrid Encoding and Mask Space Modulation

  • XIAN Jin ,
  • XU Xiaoru ,
  • XIAN Yunting ,
  • XIAN Chuhua
Expand
  • 1.School of Computer Science and Engineering，South China University of Technology，Guangzhou 510006，Guangdong，China
    2.School of Law and Humanities，Jiangsu Vocational College of Finance and Economics，Huaian 223003，Jiangsu，China
冼进（1970—），男，高级实验师，主要从事人工智能、大数据及图像处理研究。E-mail： xi88@scut.edu.cn

Received date: 2024-04-03

  Online published: 2024-10-08

Supported by

the Key-Areas R & D Program of Guangdong Province(2022B0101070001)

Fold

Abstract

Image inpainting refers to the process of filling in missing regions of an image with plausible content, which is one of the significant issues in the fields of computer vision and image processing research. Current research on image inpainting algorithms has made substantial progress. However, when dealing with complex images with large missing areas, existing algorithms still face challenges in generating high-quality complete images due to the lack of effective network structures to capture long-range dependencies and high-level semantic information in the images. To address the issue of large-scale missing image inpainting, this paper proposed an image inpainting algorithm based on hybrid encoding and mask spatial modulation. The aim is to expand the limited receptive field of image inpainting networks, effectively obtain global information from the visible regions of the image, and fully utilize the effective information from the visible regions. Firstly, a hybrid encoding network was used to extract local and global information features from the visible regions of the image. Then, a mask spatial modulation module dynamically adjusted the diversity in generating missing regions based on the size of the missing area. Finally, a method based on StyleGAN2 was used to generate complete images. Experimental results show that the proposed algorithm can effectively handle images with large-scale missing areas, generating high-quality images with diversity, and can be applied to data augmentation in visual saliency models.

Key words： image inpainting; image enhancement; hybrid encoding; mask space modulation

Cite this article

XIAN Jin , XU Xiaoru , XIAN Yunting , XIAN Chuhua . Image Inpainting Algorithm Based on Hybrid Encoding and Mask Space Modulation[J]. Journal of South China University of Technology(Natural Science), 2025 , 53(3) : 31 -39 . DOI: 10.12141/j.issn.1000-565X.240155

References

1 XIE X， PAN X， ZHANG W，et al ．A context hierarchical integrated network for medical image segmentation［J］．Computers and Electrical Engineering，2022，101：108029/1-14.
2 ALHAYANI B S A， HAMID N， ALMUKHTAR F H，et al ．Optimized video internet of things using elliptic curve cryptography based encryption and decryption［J］．Computers and Electrical Engineering，2022，101：108022/1-10.
3 MA Y， ZHAI Y， WANG R ．DeepFGS：fine-grained scalable coding for learned image compression［EB/OL］．（2022-01-04）［2024-01-02］．.
4 GOODFELLOW I， POUGET-ABADIE J， MIRZA M，et al ．Generative adversarial networks［J］．Communications of the ACM，2020，63（11）：139-144.
5 PATHAK D， KR?HENBüHL P， DONHUE J，et al ．Context encoders：feature learning by inpainting［C］∥Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：2536-2544.
6 YAN Z， LI X， LI M，et al ．Shift-Net：image inpain-ting via deep feature rearrangement［C］∥ Proceedings of the 15th European Conference on Computer Vision．Munich：Springer，2018：3-19.
7 IIZUKA S， SIMO-SERRA E， ISHIKAWA H ．Globally and locally consistent image completion［J］．ACM Transactions on Graphics，2017，36（4）：107/1-14.
8 DEMIR U， UNAL G ．Patch-based image inpainting with generative adversarial networks［EB/OL］．（2023-03-20）［2023-12-25］．.
9 YU J， LIN Z， YANG J，et al ．Generative image inpain-ting with contextual attention［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：5505-5514.
10 YANG C， LU X， LIN Z，et al ．High-resolution image inpainting using multi-scale neural patch synthesis［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：4076-4084.
11 YU J， LIN Z， YANG J，et al ．Free-form image inpainting with gated convolution［C］∥ Proceedings of 2019 IEEE/CVF International Conference on Computer Vision．Seoul：IEEE，2019：4470-4479.
12 LIU H， WAN Z， HUANG W，et al ．PD-GAN：probabilistic diverse GAN for image inpainting［C］∥Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：9367-9376.
13 ZHAO S， CUI J， SHENG Y，et al ．Large scale image completion via co-modulated generative adversarial networks［EB/OL］. （2021-03-18）［2023-12-25］．.
14 KARRAS T， LAINE S， AITTALA M，et al ．Analy-zing and improving the image quality of StyleGAN［C］∥ Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：8107-8116.
15 ZHENG H， LIN Z， LU J，et al ．Image inpainting with cascaded modulation GAN and object-aware training［C］∥ Proceedings of the 17th European Conference on Computer Vision．Tel Aviv：Springer，2022：277-296.
16 SUVOROV R， LOGACHEVA E， MASHIKHIN A，et al ．Resolution-robust large mask inpainting with Fourier convolutions［C］∥ Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision.Waikoloa：IEEE，2022：3172-3182.
17 KIM J， KIM W，OH H，et al ．Progressive contextual aggregation empowered by pixel-wise confidence scoring for image inpainting［J］．IEEE Transactions on Image Processing，2023，32：1200-1214.
18 ZHENG H， ZHANG Z， WANG Y，et al ．GCM-Net：towards effective global context modeling for image inpainting［C］∥ Proceedings of the 29th ACM International Conference on Multimedia．New York：ACM，2021：2586-2594.
19 ZHENG H， ZHANG Z， ZHANG H，et al ．Deep multi-resolution mutual learning for image inpainting［C］∥ Proceedings of the 30th ACM International Conference on Multimedia．Lisboa：ACM，2022：6359-6367.
20 FENG L， ZHU C， LONG Z，et al ．Multiplex transformed tensor decomposition for multidimensional image recovery［J］．IEEE Transactions on Image Processing，2023，32：3397-3412.
21 SIMONYAN K， ZISSERMAN A ．Very deep convolutional networks for large-scale image recognition［EB/OL］． （2015-04-10）［2023-12-25］．.
22 XIE S， GIRSHICK R， DOLLáR P，et al ．Aggregated residual transformations for deep neural networks［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：5987-5995.
23 HUANG G， LIU Z， VAN DER MAATEN L， et al ．Densely connected convolutional networks［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：2261-2269.
24 KARRAS T， LAINE S， AILA T ．A style-based gene-rator architecture for generative adversarial networks［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：4396-4405.
25 HUANG X， BELONGIE S ．Arbitrary style transfer in real-time with adaptive instance normalization［C］∥Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：1510-1519.
26 JIANG M， HUANG S， DUAN J，et al ．SALICON：saliency in context［C］∥ Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition．Boston：IEEE，2015：1072-1080.
27 SZEGEDY C， VANHOUCE V， IOFFE S，et al ．Rethinking the inception architecture for computer vision［C］∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：2818-2826.
28 ISOLA P， ZHU J Y， ZHOU T，et al ．Image-to-image translation with conditional adversarial networks［C］∥Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：5967-5976.
29 LIU G， REDA F A， SHIH K J，et al ．Image inpain-ting for irregular holes using partial convolutions［C］∥Proceedings of the 15th European Conference on Computer Vision．Munich：Springer，2018：89-105.
Options
Outlines

/

〈 〉