基于光度立体和双流特征融合网络的工业产品表面缺陷检测方法

doi:10.12141/j.issn.1000-565X.230638

摘要/Abstract

摘要：

表面缺陷检测是现代工业生产流程中的重要环节。现有的视觉缺陷检测方法一般通过对目标对象的单幅RGB或灰度图像进行分析，利用缺陷与背景之间的差异性特征实现检测，适用于目标与背景呈较大区别的对象，如金属表面的氧化、斑点缺陷检测。但单纯的RGB图像无法有效地表征主要由深度变化形成的凹坑、凸包等3维缺陷特征，最终导致漏检。为此，文中根据多方向光照成像及光度立体原理提取待测对象表面的3维几何形貌信息；接着，利用对比度金字塔融合算法对原始的多方向光照图像进行有效融合，得到增强的缺陷的2维RGB融合图像特征；然后，在多目标检测框架YOLOv5的基础上，以上述几何形貌及RGB融合图像为输入，构建一种基于双流特征融合的缺陷检测网络模型，该模型引入了空间通道注意力残差模块和门控循环单元特征融合模块，能在多个层级对不同模态特征进行有机融合，实现对表面缺陷的2维RGB及3维形貌信息的有效提取，达到同时应对2维和3维缺陷检测的目的；最后对若干典型工业产品表面缺陷进行检测实验。结果表明，文中方法在多个数据集上的平均检测准确率均超过90%，且能同时应对2维、3维缺陷的检测，检测性能优于目前的主流方法，能够适应不同工业产品表面的检测需求。

关键词: 光度立体, 缺陷检测, 深度学习, 双流特征融合

Abstract:

Surface defect detection is an important part of the modern industrial production process. The existing visual defect detection methods generally achieve detection by analyzing a single RGB or grayscale image of the target object and using differential features between the defect and the background. They are suitable for objects with a large difference between the target and the background, such as the detection of metal surface oxidation and spot defects. However, the simple RGB image cannot effectively characterize the 3D defect features such as dents and bulges, which are mainly formed by depth changes, ultimately resulting in missed detection. To this end, this paper extracted the 3D geometric appearance information of the object surface to be tested according to multi-directional light imaging and photometric stereo principle. Next, the original multi-directional light images were effectively fused using the contrast pyramid fusion algorithm to obtain the enhanced 2D RGB fusion image features of the defects. Then, on the basis of the multi-target detection framework YOLOv5, with the above geometric appearance and RGB fusion images as inputs, a defect detection network model based on dual stream feature fusion detection network model was constructed. The model introduces the spatial channel attention residual module and the gated recurrent unit (GRU) feature fusion module and is able to organically fuse the different modal features at multiple levels to realize the effective extraction of the 2D RGB and 3D appearance information of the surface defects, so as to achieve the purpose of dealing with the detection of 2D and 3D defects at the same time. Finally, the detection experiments were conducted on the surface defects of several typical industrial products. The results show that mAP of the method in the paper is above 90% on several datasets, and it can simultaneously cope with the detection of 2D and 3D defects, so the detection performance is better than that of the current mainstream methods, and it can meet the detection requirements of different industrial products.

Key words: photometric stereo, defect detection, deep learning, dual stream feature fusion

中图分类号:

TP391.41

胡广华, 涂千禧. 基于光度立体和双流特征融合网络的工业产品表面缺陷检测方法[J]. 华南理工大学学报(自然科学版), 2024, 52(10): 112-123.

HU Guanghua, TU Qianxi. Surface Defect Detection Method for Industrial Products Based on Photometric Stereo and Dual Stream Feature Fusion Network[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 112-123.

图/表 19

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

图13

表1

图14

表2

表3

表4

图15

参考文献 19

1	WOODHAM R J ．Photometric stereo：a reflectance map technique for determining surface orientation from image intensity［C］∥ Proceedings of SPIE， Image Understanding Systems and Industrial Applications Ⅰ．San Diego：SPIE， 1978：136-143．
2	SOUKUP D， HUBER-MORK R ．Convolutional neural networks for steel surface defect detection from photometric stereo images［C］∥ Proceedings of the 10th International Symposium on Visual Computing．Las Vegas：Springer-Verlag，2014：668-677．
3	CAO Y， DING B， CHEN J，et al ．Photometric-stereo-based defect detection system for metal parts［J］．Sensors，2022，22（21）：8374/1-17．
4	REN S， HE K， GIRSHICK R，et al ．Faster R-CNN：towards real-time object detection with region proposal networks［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（6）：1137-1149．
5	XIE R， ZHU Y， LUO J，et al ．Detection algorithm for bearing roller end surface defects based on improved YOLOv5n and image fusion［J］．Measurement Science and Technology，2023，34（4）：045402/1-11．
6	SAIZ F A， BARANDIARAN I， ARBELAIZ A，et al ．Photometric stereo-based defect detection system for steel components manufacturing using a deep segmentation network［J］．Sensors，2022，22（3）：882/1-14．
7	XIAO F， LI B， PENG Y，et al ． Multi-modal weights sharing and hierarchical feature fusion for RGBD salient object detection［J］． IEEE Access，2020，8：26602/1-11．
8	LI G， LIU Z， LING H ．ICNet：information conversion network for RGB-D based salient object detection［J］．IEEE Transactions on Image Processing，2020，29：4873-4884．
9	LV H， DENG B， SHI C ．Research on multi-source image fusion target detection technology based on neural network［J］．Journal of Physics：Conference Series，2021，2033：012139/1-5．
10	FU H， WANG S， DUAN P，et al ．LRAF-Net：long-range attention fusion network for visible-infrared object detection［J］．IEEE Transactions on Neural Networks and Learning Systems，2023，（2023-06-06）．Doi：10.1109/TNNLS.2023.3266452．
11	JIN X， GUO C， HE Z，et al ． FCMNet：frequency-aware cross-modality attention networks for RGB-D salient object detection［J］．Neurocomputing，2022，491：414-425．
12	金闳奇．笔记本电脑铝合金面板表面缺陷检测系统［D］．广州：广东工业大学，2022．
13	TREIBITZ T， SCHECHNER Y Y ．Turbid scene enhancement using multi-directional illumination fusion ［J］．IEEE Transactions on Image Processing，2012，21（11）：4662-4667．
14	MNIH V， HEESS N， GRAVES A，et al ． Recurrent models of visual attention［C］∥ Proceedings of the 28th Annual Conference on Neural Information Processing Systems．Montreal：Neural Information Processing Systems Foundation，2014：2204-2212．
15	WOO S， PARK J， LEE J-Y，et al ．CBAM：convolutional block attention module［C］∥ Proceedings of the 15th European Conference on Computer Vision．Munich：Springer-Verlag，2018：3-19．
16	CHUNG J， GULCEHRE C， CHO K，et al ．Empirical evaluation of gated recurrent neural networks on sequence modeling［EB/OL］．（2014-11-11）［2023-10-12］．．
17	WANG W， XIE E， SONG X，et al ．Efficient and accurate arbitrary-shaped text detection with pixel aggregation network［C］∥ Proceedings of 2019 IEEE/CVF International Conference on Computer Vision．Seoul：IEEE，2019：8439-8448．
18	YU J， JIANG Y， WANG Z，et al ．UnitBox：an advanced object detection network［C］∥ Proceedings of the 24th ACM International Conference on Multimedia． Amsterdam：ACM，2016：516-520．
19	LIU W， ANGUELOV D， ERHAN D，et al ．SSD：single shot multibox detector［C］∥ Proceedings of the 14th European Conference on Computer Vision．Amsterdam：Springer-Verlag，2016：21-37．

数据集	竹制托盘样本数			铝盖样本数			塑料瓶样本数
数据集	裂缝	脏污	孔洞	划痕	脏污	凸包	凹坑	凸包	脏污
训练集	304	308	328	257	246	214	160	157	145
测试集	121	126	85	105	88	99	63	54	40

模型	图像类型	竹制托盘缺陷的检测率/%			铝盖缺陷的检测率/%			塑料瓶缺陷的检测率/%			mAP/%
模型	图像类型	裂缝	脏污	孔洞	划痕	脏污	凸包	凹坑	凸包	脏污	竹制托盘	铝盖	塑料瓶
SSD	RGB融合图像	57.0	90.9	46.8	64.3	97.9	60.6	97.3	92.8	98.2	64.9	74.3	96.1
	散度图	81.3	44.7	83.7	54.4	5.9	80.3	98.8	98.8	59.9	69.7	46.9	85.8
	决策融合图像	60.5	88.7	73.8	60.0	78.0	67.5	99.9	95.5	94.5	74.3	68.5	96.6
Faster R-CNN	RGB融合图像	79.0	94.9	87.7	88.9	100.0	53.7	90.9	88.7	90.9	87.2	80.9	90.2
	散度图	80.9	77.0	88.5	64.0	3.4	56.8	90.6	90.9	27.5	82.1	41.4	69.7
	决策融合图像	79.9	89.4	88.1	84.3	100.0	56.8	99.4	99.4	90.9	85.8	80.4	96.6
YOLOv5	RGB融合图像	76.6	98.8	87.8	88.0	99.8	67.2	95.9	85.5	96.9	87.7	84.7	92.7
	散度图	84.3	0.9	92.4	36.8	1.0	88.4	99.4	95.8	18.8	58.9	40.1	71.3
	决策融合图像	89.3	99.2	90.2	81.7	98.3	87.7	99.4	91.3	97.3	92.9	89.2	96.0
文中模型	RGB融合图像+散度图	88.4	98.5	94.8	83.5	99.5	89.4	97.5	96.1	98.4	93.9	90.8	97.3

序号	真实标签	检测结果
序号	真实标签	SSD	Faster R-CNN	YOLOv5	文中模型
1
2
3
4
5

实验序号	M1	M2	M3	竹制托盘缺陷检测率/%			铝盖缺陷的检测率/%			塑料瓶缺陷的检测率/%			mAP/%
实验序号	M1	M2	M3	裂缝	脏污	孔洞	划痕	脏污	凸包	凹坑	凸包	脏污	竹制托盘	铝盖	塑料瓶
1				84.1	97.7	93.2	73.4	98.9	83.6	97.0	89.3	91.7	91.7	85.3	92.7
2	√			86.8	98.2	90.3	79.9	97.5	83.6	99.4	92.8	97.3	91.7	87.0	96.5
3	√	√		87.9	98.4	96.7	77.2	97.0	87.1	99.0	94.4	97.7	94.3	87.1	97.1
4	√	√	√	88.4	98.5	94.8	83.5	99.5	89.4	97.5	96.1	98.4	93.9	90.8	97.3

[1]	胡习之, 崔博非, 王琴, 等. 基于记忆泊车场景的视觉SLAM算法[J]. 华南理工大学学报(自然科学版), 2024, 52(6): 1-11.
[2]	刘昊, 元辉, 陈晨, 高伟. 基于采样的点云几何编码框架[J]. 华南理工大学学报(自然科学版), 2024, 52(6): 148-156.
[3]	杨春玲, 梁梓文. 特征域近端高维梯度下降图像压缩感知重构网络[J]. 华南理工大学学报(自然科学版), 2024, 52(3): 119-130.
[4]	郑娟毅, 董嘉豪, 张庆珏, 等. 基于残差密集网络的智能超表面信道估计算法[J]. 华南理工大学学报(自然科学版), 2024, 52(3): 102-111.
[5]	周浪, 樊坤, 瞿华, 等. 基于ECA注意力机制改进的EfficientNet-E模型的森林火灾识别[J]. 华南理工大学学报(自然科学版), 2024, 52(2): 42-49.
[6]	陈琼, 冯媛, 李志群, 等. 基于语义-视觉一致性约束的零样本图像语义分割网络[J]. 华南理工大学学报(自然科学版), 2024, 52(10): 41-50.
[7]	刘卫朋, 李旭, 任子文, 等. 多尺度残差可变形肺部CT图像配准算法[J]. 华南理工大学学报(自然科学版), 2024, 52(10): 135-145.
[8]	李方, 郭炜森, 张平, 等. 基于时空双细胞状态的轴承剩余使用寿命预测方法[J]. 华南理工大学学报(自然科学版), 2023, 51(9): 69-81.
[9]	苏锦钿, 余珊珊, 洪晓斌. 一种面向中文拼写纠错的自监督预训练方法[J]. 华南理工大学学报(自然科学版), 2023, 51(9): 90-98.
[10]	李家春, 李博文, 林伟伟. AdfNet：一种基于多样化特征的自适应深度伪造检测网络[J]. 华南理工大学学报(自然科学版), 2023, 51(9): 82-89.
[11]	郭恩强, 符锌砂. 基于特征相似性学习的抛洒物检测方法[J]. 华南理工大学学报(自然科学版), 2023, 51(6): 30-41.
[12]	赵建东, 焦岚馨, 赵志敏, 等. 考虑侧向车换道影响的理论和数据组合驱动的车辆跟驰模型[J]. 华南理工大学学报(自然科学版), 2023, 51(6): 10-19.
[13]	叶峰, 陈彪, 赖乙宗. 基于特征空间嵌入的对比知识蒸馏算法[J]. 华南理工大学学报(自然科学版), 2023, 51(5): 13-23.
[14]	赵荣超, 吴百礼, 陈祝云, 等. 多尺度时空信息融合驱动的图神经网络故障诊断方法[J]. 华南理工大学学报(自然科学版), 2023, 51(12): 42-52.
[15]	侯力玮, 王恒升, 邹浩然. 基于深度学习的玻璃基板铲起过程作用力预测[J]. 华南理工大学学报(自然科学版), 2022, 50(8): 71-81.