收稿日期: 2023-10-13
网络出版日期: 2024-03-04
基金资助
广东省自然科学基金资助项目(2022A1515010806);广州市科技计划项目(2023B01J0046)
Surface Defect Detection Method for Industrial Products Based on Photometric Stereo and Dual Stream Feature Fusion Network
Received date: 2023-10-13
Online published: 2024-03-04
Supported by
the Natural Science Foundation of Guangdong Province(2022A1515010806)
表面缺陷检测是现代工业生产流程中的重要环节。现有的视觉缺陷检测方法一般通过对目标对象的单幅RGB或灰度图像进行分析,利用缺陷与背景之间的差异性特征实现检测,适用于目标与背景呈较大区别的对象,如金属表面的氧化、斑点缺陷检测。但单纯的RGB图像无法有效地表征主要由深度变化形成的凹坑、凸包等3维缺陷特征,最终导致漏检。为此,文中根据多方向光照成像及光度立体原理提取待测对象表面的3维几何形貌信息;接着,利用对比度金字塔融合算法对原始的多方向光照图像进行有效融合,得到增强的缺陷的2维RGB融合图像特征;然后,在多目标检测框架YOLOv5的基础上,以上述几何形貌及RGB融合图像为输入,构建一种基于双流特征融合的缺陷检测网络模型,该模型引入了空间通道注意力残差模块和门控循环单元特征融合模块,能在多个层级对不同模态特征进行有机融合,实现对表面缺陷的2维RGB及3维形貌信息的有效提取,达到同时应对2维和3维缺陷检测的目的;最后对若干典型工业产品表面缺陷进行检测实验。结果表明,文中方法在多个数据集上的平均检测准确率均超过90%,且能同时应对2维、3维缺陷的检测,检测性能优于目前的主流方法,能够适应不同工业产品表面的检测需求。
胡广华 , 涂千禧 . 基于光度立体和双流特征融合网络的工业产品表面缺陷检测方法[J]. 华南理工大学学报(自然科学版), 2024 , 52(10) : 112 -123 . DOI: 10.12141/j.issn.1000-565X.230638
Surface defect detection is an important part of the modern industrial production process. The existing visual defect detection methods generally achieve detection by analyzing a single RGB or grayscale image of the target object and using differential features between the defect and the background. They are suitable for objects with a large difference between the target and the background, such as the detection of metal surface oxidation and spot defects. However, the simple RGB image cannot effectively characterize the 3D defect features such as dents and bulges, which are mainly formed by depth changes, ultimately resulting in missed detection. To this end, this paper extracted the 3D geometric appearance information of the object surface to be tested according to multi-directional light imaging and photometric stereo principle. Next, the original multi-directional light images were effectively fused using the contrast pyramid fusion algorithm to obtain the enhanced 2D RGB fusion image features of the defects. Then, on the basis of the multi-target detection framework YOLOv5, with the above geometric appearance and RGB fusion images as inputs, a defect detection network model based on dual stream feature fusion detection network model was constructed. The model introduces the spatial channel attention residual module and the gated recurrent unit (GRU) feature fusion module and is able to organically fuse the different modal features at multiple levels to realize the effective extraction of the 2D RGB and 3D appearance information of the surface defects, so as to achieve the purpose of dealing with the detection of 2D and 3D defects at the same time. Finally, the detection experiments were conducted on the surface defects of several typical industrial products. The results show that mAP of the method in the paper is above 90% on several datasets, and it can simultaneously cope with the detection of 2D and 3D defects, so the detection performance is better than that of the current mainstream methods, and it can meet the detection requirements of different industrial products.
| 1 | WOODHAM R J .Photometric stereo:a reflectance map technique for determining surface orientation from image intensity[C]∥ Proceedings of SPIE, Image Understanding Systems and Industrial Applications Ⅰ.San Diego:SPIE, 1978:136-143. |
| 2 | SOUKUP D, HUBER-MORK R .Convolutional neural networks for steel surface defect detection from photometric stereo images[C]∥ Proceedings of the 10th International Symposium on Visual Computing.Las Vegas:Springer-Verlag,2014:668-677. |
| 3 | CAO Y, DING B, CHEN J,et al .Photometric-stereo-based defect detection system for metal parts[J].Sensors,2022,22(21):8374/1-17. |
| 4 | REN S, HE K, GIRSHICK R,et al .Faster R-CNN:towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(6):1137-1149. |
| 5 | XIE R, ZHU Y, LUO J,et al .Detection algorithm for bearing roller end surface defects based on improved YOLOv5n and image fusion[J].Measurement Science and Technology,2023,34(4):045402/1-11. |
| 6 | SAIZ F A, BARANDIARAN I, ARBELAIZ A,et al .Photometric stereo-based defect detection system for steel components manufacturing using a deep segmentation network[J].Sensors,2022,22(3):882/1-14. |
| 7 | XIAO F, LI B, PENG Y,et al . Multi-modal weights sharing and hierarchical feature fusion for RGBD salient object detection[J]. IEEE Access,2020,8:26602/1-11. |
| 8 | LI G, LIU Z, LING H .ICNet:information conversion network for RGB-D based salient object detection[J].IEEE Transactions on Image Processing,2020,29:4873-4884. |
| 9 | LV H, DENG B, SHI C .Research on multi-source image fusion target detection technology based on neural network[J].Journal of Physics:Conference Series,2021,2033:012139/1-5. |
| 10 | FU H, WANG S, DUAN P,et al .LRAF-Net:long-range attention fusion network for visible-infrared object detection[J].IEEE Transactions on Neural Networks and Learning Systems,2023,(2023-06-06).Doi:10.1109/TNNLS.2023.3266452. |
| 11 | JIN X, GUO C, HE Z,et al . FCMNet:frequency-aware cross-modality attention networks for RGB-D salient object detection[J].Neurocomputing,2022,491:414-425. |
| 12 | 金闳奇 .笔记本电脑铝合金面板表面缺陷检测系统 [D].广州:广东工业大学,2022. |
| 13 | TREIBITZ T, SCHECHNER Y Y .Turbid scene enhancement using multi-directional illumination fusion [J].IEEE Transactions on Image Processing,2012,21(11):4662-4667. |
| 14 | MNIH V, HEESS N, GRAVES A,et al . Recurrent models of visual attention[C]∥ Proceedings of the 28th Annual Conference on Neural Information Processing Systems.Montreal:Neural Information Processing Systems Foundation,2014:2204-2212. |
| 15 | WOO S, PARK J, LEE J-Y,et al .CBAM:convolutional block attention module[C]∥ Proceedings of the 15th European Conference on Computer Vision.Munich:Springer-Verlag,2018:3-19. |
| 16 | CHUNG J, GULCEHRE C, CHO K,et al .Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL].(2014-11-11)[2023-10-12]. . |
| 17 | WANG W, XIE E, SONG X,et al .Efficient and accurate arbitrary-shaped text detection with pixel aggregation network[C]∥ Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Seoul:IEEE,2019:8439-8448. |
| 18 | YU J, JIANG Y, WANG Z,et al .UnitBox:an advanced object detection network[C]∥ Proceedings of the 24th ACM International Conference on Multimedia. Amsterdam:ACM,2016:516-520. |
| 19 | LIU W, ANGUELOV D, ERHAN D,et al .SSD:single shot multibox detector[C]∥ Proceedings of the 14th European Conference on Computer Vision.Amsterdam:Springer-Verlag,2016:21-37. |
/
| 〈 |
|
〉 |