Surface Defect Detection Method for Industrial Products Based on Photometric Stereo and Dual Stream Feature Fusion Network

doi:10.12141/j.issn.1000-565X.230638

Abstract

Abstract:

Surface defect detection is an important part of the modern industrial production process. The existing visual defect detection methods generally achieve detection by analyzing a single RGB or grayscale image of the target object and using differential features between the defect and the background. They are suitable for objects with a large difference between the target and the background, such as the detection of metal surface oxidation and spot defects. However, the simple RGB image cannot effectively characterize the 3D defect features such as dents and bulges, which are mainly formed by depth changes, ultimately resulting in missed detection. To this end, this paper extracted the 3D geometric appearance information of the object surface to be tested according to multi-directional light imaging and photometric stereo principle. Next, the original multi-directional light images were effectively fused using the contrast pyramid fusion algorithm to obtain the enhanced 2D RGB fusion image features of the defects. Then, on the basis of the multi-target detection framework YOLOv5, with the above geometric appearance and RGB fusion images as inputs, a defect detection network model based on dual stream feature fusion detection network model was constructed. The model introduces the spatial channel attention residual module and the gated recurrent unit (GRU) feature fusion module and is able to organically fuse the different modal features at multiple levels to realize the effective extraction of the 2D RGB and 3D appearance information of the surface defects, so as to achieve the purpose of dealing with the detection of 2D and 3D defects at the same time. Finally, the detection experiments were conducted on the surface defects of several typical industrial products. The results show that mAP of the method in the paper is above 90% on several datasets, and it can simultaneously cope with the detection of 2D and 3D defects, so the detection performance is better than that of the current mainstream methods, and it can meet the detection requirements of different industrial products.

Key words: photometric stereo, defect detection, deep learning, dual stream feature fusion

CLC Number:

TP391.41

HU Guanghua, TU Qianxi. Surface Defect Detection Method for Industrial Products Based on Photometric Stereo and Dual Stream Feature Fusion Network[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 112-123.

Figures/Tables 19

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Table 1

Fig.14

Table 2

Table 3

Table 4

Fig.15

References 19

1	WOODHAM R J ．Photometric stereo：a reflectance map technique for determining surface orientation from image intensity［C］∥ Proceedings of SPIE， Image Understanding Systems and Industrial Applications Ⅰ．San Diego：SPIE， 1978：136-143．
2	SOUKUP D， HUBER-MORK R ．Convolutional neural networks for steel surface defect detection from photometric stereo images［C］∥ Proceedings of the 10th International Symposium on Visual Computing．Las Vegas：Springer-Verlag，2014：668-677．
3	CAO Y， DING B， CHEN J，et al ．Photometric-stereo-based defect detection system for metal parts［J］．Sensors，2022，22（21）：8374/1-17．
4	REN S， HE K， GIRSHICK R，et al ．Faster R-CNN：towards real-time object detection with region proposal networks［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（6）：1137-1149．
5	XIE R， ZHU Y， LUO J，et al ．Detection algorithm for bearing roller end surface defects based on improved YOLOv5n and image fusion［J］．Measurement Science and Technology，2023，34（4）：045402/1-11．
6	SAIZ F A， BARANDIARAN I， ARBELAIZ A，et al ．Photometric stereo-based defect detection system for steel components manufacturing using a deep segmentation network［J］．Sensors，2022，22（3）：882/1-14．
7	XIAO F， LI B， PENG Y，et al ． Multi-modal weights sharing and hierarchical feature fusion for RGBD salient object detection［J］． IEEE Access，2020，8：26602/1-11．
8	LI G， LIU Z， LING H ．ICNet：information conversion network for RGB-D based salient object detection［J］．IEEE Transactions on Image Processing，2020，29：4873-4884．
9	LV H， DENG B， SHI C ．Research on multi-source image fusion target detection technology based on neural network［J］．Journal of Physics：Conference Series，2021，2033：012139/1-5．
10	FU H， WANG S， DUAN P，et al ．LRAF-Net：long-range attention fusion network for visible-infrared object detection［J］．IEEE Transactions on Neural Networks and Learning Systems，2023，（2023-06-06）．Doi：10.1109/TNNLS.2023.3266452．
11	JIN X， GUO C， HE Z，et al ． FCMNet：frequency-aware cross-modality attention networks for RGB-D salient object detection［J］．Neurocomputing，2022，491：414-425．
12	金闳奇．笔记本电脑铝合金面板表面缺陷检测系统［D］．广州：广东工业大学，2022．
13	TREIBITZ T， SCHECHNER Y Y ．Turbid scene enhancement using multi-directional illumination fusion ［J］．IEEE Transactions on Image Processing，2012，21（11）：4662-4667．
14	MNIH V， HEESS N， GRAVES A，et al ． Recurrent models of visual attention［C］∥ Proceedings of the 28th Annual Conference on Neural Information Processing Systems．Montreal：Neural Information Processing Systems Foundation，2014：2204-2212．
15	WOO S， PARK J， LEE J-Y，et al ．CBAM：convolutional block attention module［C］∥ Proceedings of the 15th European Conference on Computer Vision．Munich：Springer-Verlag，2018：3-19．
16	CHUNG J， GULCEHRE C， CHO K，et al ．Empirical evaluation of gated recurrent neural networks on sequence modeling［EB/OL］．（2014-11-11）［2023-10-12］．．
17	WANG W， XIE E， SONG X，et al ．Efficient and accurate arbitrary-shaped text detection with pixel aggregation network［C］∥ Proceedings of 2019 IEEE/CVF International Conference on Computer Vision．Seoul：IEEE，2019：8439-8448．
18	YU J， JIANG Y， WANG Z，et al ．UnitBox：an advanced object detection network［C］∥ Proceedings of the 24th ACM International Conference on Multimedia． Amsterdam：ACM，2016：516-520．
19	LIU W， ANGUELOV D， ERHAN D，et al ．SSD：single shot multibox detector［C］∥ Proceedings of the 14th European Conference on Computer Vision．Amsterdam：Springer-Verlag，2016：21-37．

数据集	竹制托盘样本数			铝盖样本数			塑料瓶样本数
数据集	裂缝	脏污	孔洞	划痕	脏污	凸包	凹坑	凸包	脏污
训练集	304	308	328	257	246	214	160	157	145
测试集	121	126	85	105	88	99	63	54	40

模型	图像类型	竹制托盘缺陷的检测率/%			铝盖缺陷的检测率/%			塑料瓶缺陷的检测率/%			mAP/%
模型	图像类型	裂缝	脏污	孔洞	划痕	脏污	凸包	凹坑	凸包	脏污	竹制托盘	铝盖	塑料瓶
SSD	RGB融合图像	57.0	90.9	46.8	64.3	97.9	60.6	97.3	92.8	98.2	64.9	74.3	96.1
	散度图	81.3	44.7	83.7	54.4	5.9	80.3	98.8	98.8	59.9	69.7	46.9	85.8
	决策融合图像	60.5	88.7	73.8	60.0	78.0	67.5	99.9	95.5	94.5	74.3	68.5	96.6
Faster R-CNN	RGB融合图像	79.0	94.9	87.7	88.9	100.0	53.7	90.9	88.7	90.9	87.2	80.9	90.2
	散度图	80.9	77.0	88.5	64.0	3.4	56.8	90.6	90.9	27.5	82.1	41.4	69.7
	决策融合图像	79.9	89.4	88.1	84.3	100.0	56.8	99.4	99.4	90.9	85.8	80.4	96.6
YOLOv5	RGB融合图像	76.6	98.8	87.8	88.0	99.8	67.2	95.9	85.5	96.9	87.7	84.7	92.7
	散度图	84.3	0.9	92.4	36.8	1.0	88.4	99.4	95.8	18.8	58.9	40.1	71.3
	决策融合图像	89.3	99.2	90.2	81.7	98.3	87.7	99.4	91.3	97.3	92.9	89.2	96.0
文中模型	RGB融合图像+散度图	88.4	98.5	94.8	83.5	99.5	89.4	97.5	96.1	98.4	93.9	90.8	97.3

序号	真实标签	检测结果
序号	真实标签	SSD	Faster R-CNN	YOLOv5	文中模型
1
2
3
4
5

实验序号	M1	M2	M3	竹制托盘缺陷检测率/%			铝盖缺陷的检测率/%			塑料瓶缺陷的检测率/%			mAP/%
实验序号	M1	M2	M3	裂缝	脏污	孔洞	划痕	脏污	凸包	凹坑	凸包	脏污	竹制托盘	铝盖	塑料瓶
1				84.1	97.7	93.2	73.4	98.9	83.6	97.0	89.3	91.7	91.7	85.3	92.7
2	√			86.8	98.2	90.3	79.9	97.5	83.6	99.4	92.8	97.3	91.7	87.0	96.5
3	√	√		87.9	98.4	96.7	77.2	97.0	87.1	99.0	94.4	97.7	94.3	87.1	97.1
4	√	√	√	88.4	98.5	94.8	83.5	99.5	89.4	97.5	96.1	98.4	93.9	90.8	97.3

[1]	HU Xizhi, CUI Bofei, WANG Qin, et al. Visual SLAM Algorithm Based on Memory Parking Scene [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(6): 1-11.
[2]	LIU Hao, YUAN Hui, CHEN Chen, GAO Wei. Point Cloud Geometry Coding Framework Based on Sampling [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(6): 148-156.
[3]	YANG Chunling, LIANG Ziwen. Feature-Domain Proximal High-Dimensional Gradient Descent Network for Image Compressed Sensing [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(3): 119-130.
[4]	ZHENG Juanyi, DONG Jiahao, ZHANG Qingjue, et al. Reconfigurable Intelligence Surface Channel Estimation Algorithm Based on RDN [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(3): 102-111.
[5]	ZHOU Lang, FAN Kun, QU Hua, et al. Forest Fire Recognition by Improved EfficientNet-E Model Based on ECA Attention Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(2): 42-49.
[6]	CHEN Qiong, FENG Yuan, LI Zhiqun, et al. Semantic-Visual Consistency Constraint Network for Zero-Shot Image Semantic Segmentation [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 41-50.
[7]	LIU Weipeng, LI Xu, REN Ziwen, et al. Algorithm for Registration of Multiscale Residual Deformable Lung CT Images #br# [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 135-145.
[8]	LI Fang, GUO Weisen, ZHANG Ping, et al.. Prediction Technique for Remaining Useful Life of Bearing Based on Spatial-Temporal Dual Cell State [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 69-81.
[9]	SU Jindian, YU Shanshan, HONG Xiaobin. A Self-Supervised Pre-Training Method for Chinese Spelling Correction [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 90-98.
[10]	LI Jiachun, LI Bowen, LIN Weiwei. AdfNet: An Adaptive Deep Forgery Detection Network Based on Diverse Features [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 82-89.
[11]	GUO Enqiang, FU Xinsha. Dropped Object Detection Method Based on Feature Similarity Learning [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(6): 30-41.
[12]	ZHAO Jiandong, JIAO Lanxin, ZHAO Zhimin, et al. A Car-Following Model Driven by Combination of Theory and Data Considering Effects of Lane Change of Side Cars [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(6): 10-19.
[13]	YE Feng, CHEN Biao, LAI Yizong. Contrastive Knowledge Distillation Method Based on Feature Space Embedding [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 13-23.
[14]	ZHAO Rongchao, WU Baili, CHEN Zhuyun, et al. Graph Neural Network for Fault Diagnosis with Multi-Scale Time-Spatial Information Fusion Mechanism [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(12): 42-52.
[15]	HOU Liwei, WANG Hengsheng, ZOU Haoran. Deep Learning-Based Prediction of Contact Force in the Process of Shoveling Up Glass Subtrate [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(8): 71-81.