A Novel Generative Image Inpainting Model with Dense Gated Convolutional Network

Xiaoxuan Ma; Yibo Deng; Lei Zhang; Zhiwen Li

doi:10.15837/ijccc.2023.2.5088

Authors

Xiaoxuan Ma School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, Beijing, China
Yibo Deng School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, China
Lei Zhang School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, Beijing, China
Zhiwen Li School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, Beijing, China

DOI:

https://doi.org/10.15837/ijccc.2023.2.5088

Keywords:

Densely Connected Convolutional Networks, Gated Convolution; image inpainting, Generative Adversarial Networks

Abstract

Damaged image inpainting is one of the hottest research fields in computer image processing. The development of deep learning, especially Convolutional Neural Network (CNN), has significantly enhanced the effect of image inpainting. However, the direct connection between convolution layers may increase the risk of gradient disappearance or overfitting during training process. In addition, pixel artifacts or visual inconsistencies may occur if the damaged area is inpainted directly. To solve the above problems, we propose a novel Dense Gated Convolutional Network (DGCN) for generative image inpainting by modifying the gated convolutional network structure in this paper. Firstly, Holistically-nested edge detector (HED) is utilized to predict the edge information of the missing areas to assist the subsequent inpainting task to reduce the generation of artifacts. Then, dense connections are added to the generative network to reduce the network parameters while reducing the risk of instability in the training process. Finally, the experimental results on CelebA and Places2 datasets show that the proposed model achieves better inpainting results in terms of PSNR, SSIM and visual effects compared with other classical image inpainting models. DGCN has the common advantages of gated convolution and dense connection, which can reduce network parameters and improve the inpainting effect of the network.

References

Efros, A.A.; Leung, T.K. (1999). Texture synthesis by non-parametric sampling, proc.ieee int.conf.on computer vision, 1999.

https://doi.org/10.1109/ICCV.1999.790383

Kwatra, V.; Essa, I.A.; Bobick, A.F.; Kwatra, N. (2005). Texture optimization for example-based synthesis, ACM Transactions on Graphics (TOG), 24(3),795-802,2005.

https://doi.org/10.1145/1073204.1073263

Ding D.; Ram S.; Rodríguez J.J. (2019). Image inpainting using nonlocal texture matching and nonlinear filtering, IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 1705-1719, April 2019.

https://doi.org/10.1109/TIP.2018.2880681

Erkan, U.; Enginoglu, S.; Dang, N. (2019). An Iterative Image Inpainting Method Based on Similarity of Pixels Values, IEEE The 6th International Conference on Electrical and Electronics Engineering ICEEE 2019, 2019.

https://doi.org/10.1109/ICEEE2019.2019.00028

Wang, H.; Jiang, L.; Liang, R.; Li, X.X. (2017). Exemplar-based image inpainting using structure consistent patch matching, Neurocomputing, 269(dec.20), 401-410.

https://doi.org/10.1007/978-3-319-23989-7_41

Song, Y.; Yang, C.; Lin, Z.; Liu, X.; Huang, Q.; Li, H. (2017). Contextual-based Image Inpainting: Infer, Match, and Translate, Proceedings of the European Conference on Computer Vision, pages 3-19, 2017.

https://doi.org/10.1007/978-3-030-01216-8_1

Cheng, Y.; Wan, Y.; Sima, Y.; Zhang, Y.; Hu, S.; Wu, S. (2022). Text Detection of Transformer Based on Deep Learning Algorithm, Tehnički vjesnik, 29 (3), 861-866.

https://doi.org/10.17559/TV-20211027110610

Lv, B.; Gao, X.; Feng, S.; Yuan, J. (2022). Deep Learning Detection Algorithm for Surface Defects of Automobile Door Seals, Tehnički vjesnik, 29 (5), 1499-1506.

https://doi.org/10.17559/TV-20211219032823

Ma, X.; Li, Z.; Zhang, L. (2022). An Improved ResNet-50 for Garbage Image Classification, Tehnički vjesnik, 29 (5), 1552-1559.

https://doi.org/10.17559/TV-20220420124810

Lee, J.M.; Jung, I.H.; Hwang, K. (2022). Classification of beef by using artificial intelligence, Journal of Logistics, Informatics and Service Science, Vol. 9, No. 3, pp. 271-283.

Wang, J.; Chen, Y.; Qiu, S.; Cui, Q. (2021). Cuckoo Search Optimized Integrated Framework Based on Feature Clustering and Deep Learning for Daily Stock Price Forecasting, Economic Computation And Economic Cybernetics Studies And Research, Vol. 55(3), pg. 55-70, DOI: 10.24818/18423264/55.3.21.04.

https://doi.org/10.24818/18423264/55.3.21.04

Lu, W.; Li, J.; Wang, J.; Wu, S. (2022). A Novel Model for Stock Closing Price Prediction Using CNN-Attention-GRU-Attention, Economic Computation And Economic Cybernetics Studies And Research, Vol. 56(3), pg. 251-264, DOI: 10.24818/18423264/56.3.22.16.

https://doi.org/10.24818/18423264/56.3.22.16

Nath, B., Kumbhar, C., Khoa, B.T. (2022). A study on approaches to neural machine translation, Journal of Logistics, Informatics and Service Science, Vol. 9, No. 3, pp. 271-283.

Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; et al. (2014). Generative Adversarial Nets, Neural Information Processing Systems, MIT Press,2014.

Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, A. A. (2016). Context Encoders: Feature Learning by Inpainting, IEEE, 10.1109/CVPR.2016.278, 2016.

https://doi.org/10.1109/CVPR.2016.278

Wang, Y.; Tao, X.; Qi, X.; Shen, X.; Jia, J. (2018). Image inpainting via generative multi-column convolutional neural networks, , 2018.

Guilin Liu; Fitsum A. Reda; Kevin J. Shih; Ting-Chun Wang; et al (2018). Image inpainting for irregular holes using partial convolutions, Proceedings of the European Conference on Computer Vision (ECCV), pages 85-100,2018.

Yu, J.; Lin, Z.; Yang, J.; Shen, X.; Lu, X.; Huang, T. (2018). Free-Form Image Inpainting With Gated Convolution, 2019 IEEE/CVF International Conference on Computer Vision, pp. 4470-4479, 2018.

https://doi.org/10.1109/ICCV.2019.00457

He, J.; Zhang, S.; Yang, M.; Shan, Y.; Huang, T. (2019). BDCN: Bi-Directional Cascade Network for Perceptual Edge Detection, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3823-3832, 2019.

https://doi.org/10.1109/CVPR.2019.00395

Canny, J. (1986). A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6), 679-698,1986.

https://doi.org/10.1109/TPAMI.1986.4767851

Liu, Y.; Cheng, M.M.; Hu, X.; Wang, K.; Bai, X. (2016). Richer convolutional features for edge detection, IEEE Computer Society, 2016.

https://doi.org/10.1109/CVPR.2017.622

Xie, S.; and Z. Tu. (2015). Holistically-Nested Edge Detection, 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1395-1403, 2015.

https://doi.org/10.1109/ICCV.2015.164

Poma, X.S.; Riba, E.; Sappa, A.D. (2019). Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection, , 10.48550/arXiv.1909.01955.

Arimoto M.; Hara J.; Watanabe H. (2021). An Image Inpainting Method Considering Edge Connectivity of Defects, 2021 IEEE 10th Global Conference on Consumer Electronics (GCCE), pp. 101-102, 2021.

https://doi.org/10.1109/GCCE53005.2021.9621917

Yuantao Chen; Runlong Xia; Ke Zou; Kai Yang (2023). E2I: Generative Inpainting From Edge to Image, Journal of Visual Communication and Image Representation, Volume 91,2023,103776, ISSN 1047-3203.

https://doi.org/10.1016/j.jvcir.2023.103776

Xu, S.; Liu,D.; Xiong Z. (2021). FFTI: Image inpainting algorithm via features fusion and twosteps inpainting, IEEE Transactions on Circuits and Systems for Video Technology, 31(),1308- 1322,2021.

https://doi.org/10.1109/TCSVT.2020.3001267

Haiyan Li; Yingqing Song; Haijiang Li; Zhengyu Wang (2023). Semantic prior-driven fused contextual transformation network for image inpainting, Journal of Visual Communication and Image Representation, Volume 91,2023,103777, ISSN 1047-3203.

https://doi.org/10.1016/j.jvcir.2023.103777

He, K.; Zhang, X.; Ren, S.; Sun, J. (2016). Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778,2016.

https://doi.org/10.1109/CVPR.2016.90

Huang, G.; Liu, Z.; Laurens, V.; Weinberger, K.Q. (2016). Densely connected convolutional networks, IEEE Computer Society, 2016.

https://doi.org/10.1109/CVPR.2017.243

Pleiss, G.; Chen, D.; Huang, G.; Li, T.; Laurens, V.; Weinberger, K.Q. (2017). Memory-efficient implementation of densenets, , 2017.

Yu, J.; Lin, Z.; Yang, J.; Shen, X.; Lu, X.; Huang, T.S. (2018). Generative image inpainting with contextual attention, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5505-5514, 2018.

https://doi.org/10.1109/CVPR.2018.00577

Iizuka, S.; Simo-Serra, E.; Ishikawa, H. (2017). Globally and locally consistent image completion, ACM Transactions on Graphics (TOG), 36(4CD), 107.1-107.14,2017.

https://doi.org/10.1145/3072959.3073659

Nazeri, K.; Ng, E.; Joseph, T.; Qureshi, F.Z.; Ebrahimi, M. (2019). Edgeconnect: generative image inpainting with adversarial edge learning, , 2019.

Nair, V.; Hinton, G.E. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines Vinod Nair, International Conference on International Conference on Machine Learning, Omni press,2010.

Zhou, W.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. (2004). Image quality assessment: from error visibility to structural similarity, IEEE Trans Image Process, 13(4),2004.

https://doi.org/10.1109/TIP.2003.819861

Ioffe, S.; Szegedy, C. (2015). Batch normalization: accelerating deep network training by reducing internal covariate shift, JMLR, org,2015.