Image Inpainting Algorithm Based on Hybrid Encoding and Mask Space Modulation

doi:10.12141/j.issn.1000-565X.240155

Abstract

Abstract:

Image inpainting refers to the process of filling in missing regions of an image with plausible content, which is one of the significant issues in the fields of computer vision and image processing research. Current research on image inpainting algorithms has made substantial progress. However, when dealing with complex images with large missing areas, existing algorithms still face challenges in generating high-quality complete images due to the lack of effective network structures to capture long-range dependencies and high-level semantic information in the images. To address the issue of large-scale missing image inpainting, this paper proposed an image inpainting algorithm based on hybrid encoding and mask spatial modulation. The aim is to expand the limited receptive field of image inpainting networks, effectively obtain global information from the visible regions of the image, and fully utilize the effective information from the visible regions. Firstly, a hybrid encoding network was used to extract local and global information features from the visible regions of the image. Then, a mask spatial modulation module dynamically adjusted the diversity in generating missing regions based on the size of the missing area. Finally, a method based on StyleGAN2 was used to generate complete images. Experimental results show that the proposed algorithm can effectively handle images with large-scale missing areas, generating high-quality images with diversity, and can be applied to data augmentation in visual saliency models.

Key words: image inpainting, image enhancement, hybrid encoding, mask space modulation

CLC Number:

TP391.41

XIAN Jin, XU Xiaoru, XIAN Yunting, XIAN Chuhua. Image Inpainting Algorithm Based on Hybrid Encoding and Mask Space Modulation[J]. Journal of South China University of Technology(Natural Science Edition), 2025, 53(3): 31-39.

Figures/Tables 7

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Fig.5

Table 2

References 29

1	XIE X， PAN X， ZHANG W，et al ．A context hierarchical integrated network for medical image segmentation［J］．Computers and Electrical Engineering，2022，101：108029/1-14.
2	ALHAYANI B S A， HAMID N， ALMUKHTAR F H，et al ．Optimized video internet of things using elliptic curve cryptography based encryption and decryption［J］．Computers and Electrical Engineering，2022，101：108022/1-10.
3	MA Y， ZHAI Y， WANG R ．DeepFGS：fine-grained scalable coding for learned image compression［EB/OL］．（2022-01-04）［2024-01-02］．.
4	GOODFELLOW I， POUGET-ABADIE J， MIRZA M，et al ．Generative adversarial networks［J］．Communications of the ACM，2020，63（11）：139-144.
5	PATHAK D， KRÄHENBÜHL P， DONHUE J，et al ．Context encoders：feature learning by inpainting［C］∥Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：2536-2544.
6	YAN Z， LI X， LI M，et al ．Shift-Net：image inpain-ting via deep feature rearrangement［C］∥ Proceedings of the 15th European Conference on Computer Vision．Munich：Springer，2018：3-19.
7	IIZUKA S， SIMO-SERRA E， ISHIKAWA H ．Globally and locally consistent image completion［J］．ACM Transactions on Graphics，2017，36（4）：107/1-14.
8	DEMIR U， UNAL G ．Patch-based image inpainting with generative adversarial networks［EB/OL］．（2023-03-20）［2023-12-25］．.
9	YU J， LIN Z， YANG J，et al ．Generative image inpain-ting with contextual attention［C］∥ Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：5505-5514.
10	YANG C， LU X， LIN Z，et al ．High-resolution image inpainting using multi-scale neural patch synthesis［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：4076-4084.
11	YU J， LIN Z， YANG J，et al ．Free-form image inpainting with gated convolution［C］∥ Proceedings of 2019 IEEE/CVF International Conference on Computer Vision．Seoul：IEEE，2019：4470-4479.
12	LIU H， WAN Z， HUANG W，et al ．PD-GAN：probabilistic diverse GAN for image inpainting［C］∥Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：9367-9376.
13	ZHAO S， CUI J， SHENG Y，et al ．Large scale image completion via co-modulated generative adversarial networks［EB/OL］. （2021-03-18）［2023-12-25］．.
14	KARRAS T， LAINE S， AITTALA M，et al ．Analy-zing and improving the image quality of StyleGAN［C］∥ Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：8107-8116.
15	ZHENG H， LIN Z， LU J，et al ．Image inpainting with cascaded modulation GAN and object-aware training［C］∥ Proceedings of the 17th European Conference on Computer Vision．Tel Aviv：Springer，2022：277-296.
16	SUVOROV R， LOGACHEVA E， MASHIKHIN A，et al ．Resolution-robust large mask inpainting with Fourier convolutions［C］∥ Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision.Waikoloa：IEEE，2022：3172-3182.
17	KIM J， KIM W，OH H，et al ．Progressive contextual aggregation empowered by pixel-wise confidence scoring for image inpainting［J］．IEEE Transactions on Image Processing，2023，32：1200-1214.
18	ZHENG H， ZHANG Z， WANG Y，et al ．GCM-Net：towards effective global context modeling for image inpainting［C］∥ Proceedings of the 29th ACM International Conference on Multimedia．New York：ACM，2021：2586-2594.
19	ZHENG H， ZHANG Z， ZHANG H，et al ．Deep multi-resolution mutual learning for image inpainting［C］∥ Proceedings of the 30th ACM International Conference on Multimedia．Lisboa：ACM，2022：6359-6367.
20	FENG L， ZHU C， LONG Z，et al ．Multiplex transformed tensor decomposition for multidimensional image recovery［J］．IEEE Transactions on Image Processing，2023，32：3397-3412.
21	SIMONYAN K， ZISSERMAN A ．Very deep convolutional networks for large-scale image recognition［EB/OL］．（2015-04-10）［2023-12-25］．.
22	XIE S， GIRSHICK R， DOLLÁR P，et al ．Aggregated residual transformations for deep neural networks［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：5987-5995.
23	HUANG G， LIU Z， VAN DER MAATEN L， et al ．Densely connected convolutional networks［C］∥ Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：2261-2269.
24	KARRAS T， LAINE S， AILA T ．A style-based gene-rator architecture for generative adversarial networks［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：4396-4405.
25	HUANG X， BELONGIE S ．Arbitrary style transfer in real-time with adaptive instance normalization［C］∥Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：1510-1519.
26	JIANG M， HUANG S， DUAN J，et al ．SALICON：saliency in context［C］∥ Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition．Boston：IEEE，2015：1072-1080.
27	SZEGEDY C， VANHOUCE V， IOFFE S，et al ．Rethinking the inception architecture for computer vision［C］∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：2818-2826.
28	ISOLA P， ZHU J Y， ZHOU T，et al ．Image-to-image translation with conditional adversarial networks［C］∥Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：5967-5976.
29	LIU G， REDA F A， SHIH K J，et al ．Image inpain-ting for irregular holes using partial convolutions［C］∥Proceedings of the 15th European Conference on Computer Vision．Munich：Springer，2018：89-105.

算法	D_FI	S_LPIP	s_PID/%	s_UID/%
Pix2Pix	40.81	0.40	0.00	0.00
PConv	45.55	0.39	0.00	0.00
PDGAN	32.83	0.38	0.00	0.00
CO-MOD-GAN	10.95	0.25	4.68	17.73
文献［17］算法	9.90	0.24	7.23	21.32
文中［20］算法	9.91	0.24	7.23	21.33
文中算法	9.92	0.24	7.24	21.30

方法	D_FI	S_LPIP	s_PID/%	s_UID/%
CNN+StyleGAN2+MSM	24.92	0.39	6.43	18.52
DCM+StyleGAN2	32.23	0.35	6.32	19.64
DCM+StyleGAN2+MSM	9.92	0.24	7.24	21.30

[1]	LI Haiyan, YIN Haolin, LI Peng, et al.. Image Inpainting Algorithm Based on Dense Feature Reasoning and Mix Loss Function [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 99-109.
[2]	YU Ying, HE Penghao, XU Chaoyue . Image Inpainting via Residual Attention Fusion and Gated Information Distillation [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(12): 49-59.
[3]	HU Guanghua, WANG Ning, HE Wenliang, et al. Unsupervised Surface Defect Detection Method Based on Image Inpainting [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(7): 76-85,124.
[4]	MA Rong-gui WANG Wei-xing LIU Wei ZHANG Yi XU Lian. A Haze Removal Algorithm Combining Fractional Differential,Dark Channel Prior and Retinex [J]. Journal of South China University of Technology (Natural Science Edition), 2016, 44(9): 16-23.
[5]	Wang Wei-ning Liu Jian-cong Xu Xiang-min Jiang Yi-zi Wang Li. Aesthetic Enhancement of Images Based on Photography Composition Guidelines [J]. Journal of South China University of Technology (Natural Science Edition), 2015, 43(5): 51-58.
[6]	Wu Yi - quan Yin Jun Dai Yi - mian . Image Enhancement in NSCT Domain Based on Fuzzy Sets and Artificial Bee Colony Optimization [J]. Journal of South China University of Technology (Natural Science Edition), 2015, 43(1): 59-65.
[7]	Qin Chuan- bo Du Qi- liang Tian Lian- fang Zhang Qin. Cell Segmentation in the Process of Cell Microinjection Based on Variation Level Set [J]. Journal of South China University of Technology (Natural Science Edition), 2014, 42(3): 15-20,34.
[8]	Yu Jin-dong Zhang Xian-min Chen Zhong. Retinex Enhancement Algorithm with Scale Optimization andMicro Vision Image Enhancement [J]. Journal of South China University of Technology (Natural Science Edition), 2013, 41(2): 7-11.
[9]	Zhang Yi Pu Yi-fei Zhou Ji-liu. Construction of Fractional Derivation Masks and Corresponding Filter [J]. Journal of South China University of Technology (Natural Science Edition), 2011, 39(7): 95-101.