Multi-Object Recognition and 6-DoF Pose Estimation Based on Synthetic Datasets

doi:10.12141/j.issn.1000-565X.230327

Abstract

Abstract:

Multi-object recognition and 6-DoF (degree of freedom) pose estimation are the key to achieve automatic sorting of robots in the state of unordered stacking of materials. In recent years, methods based on deep neural networks have received much attention in the multi-object recognition and 6-DoF pose estimation fields. Such methods rely on a large number of training samples, however, the collection and labeling of samples is time-consuming and laborious, which limits its application. In addition, when the imaging conditions are poor and the targets are occluded by each other, the existing pose estimation methods cannot guarantee the reliability of the results, resulting in grasping failures. To this end, this paper presented a method for target recognition, segmentation and pose estimation based on synthetic data samples. Firstly, multi-view RGB-D synthetic images of virtual scenes were generated using 3D graphics programming tools based on the 3D geometric models of the target objects, and then style transfer and noise enhancement was performed, respectively, on the generated RGB images and the depth images to improve their realism, so that they are suited for the detection in real scenes. Next, the YOLOv7-mask instance segmentation model was trained with synthetic datasets and tested by real data. The results demonstrate the effectiveness of the proposed method. Secondly, the ES6D model was utilized to estimate target poses based on the segmentation results, and an online posture evaluation method was proposed to automatically filter out severely distorted estimation results. Finally, a pose estimation correction strategy based on active vision technique was proposed to guide the robot arm to move to a new viewpoint for re-detection, which can effectively solve the problem of pose estimation deviation caused by occlusion. The above methods have been verified on a self-built 6-DoF industrial robot vision sorting system. The experimental results show that the proposed algorithm can well meet the requirements of recognition and 6-DoF posture estimation of common workpieces in complex environments.

Key words: object recognition, position measurement, 6-DoF pose estimation, robot automatic sorting, RGB-D image

CLC Number:

TP391.41

HU Guanghua, OU Meitong, LI Zhendong. Multi-Object Recognition and 6-DoF Pose Estimation Based on Synthetic Datasets[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(4): 42-50.

Figures/Tables 18

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Fig.8

Table 2

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Table 3

Fig.15

References 18

1	翟敬梅，黄乐．机器人无序分拣技术研究［J］．包装工程艺术版，2022，43（8）：66-75.
	ZHAI Jing-mei， HUANG Le ．Review of unordered picking technology for robots［J］．Packaging Engineering，2022，43（8）：66-75.
2	王高，陈晓鸿，柳宁，等．一种基于视角选择经验增强算法的机器人抓取策略［J］．华南理工大学学报（自然科学版），2022，50（9）：126-137.
	WANG Gao， CHEN Xiaohong， LIU Ning，et al ．A robot grasping policy based on viewpoint selection experience enhancement algorithm［J］．Journal of South China University of Technology （Natural Science Edition），2022，50（9）：126-137.
3	HINTERSTOISSER S， LEPETIT V， ILIC S，et al ．Model based training，detection and pose estimation of texture-less 3D objects in heavily cluttered scenes［C］∥Proceedings of the 11th Asian Conference on Computer Vision．Berlin/Heidelberg：Springer，2013：548-562.
4	DROST B， ULRICH M， NAVAB N，et al ．Model globally，match locally：efficient and robust 3D object recognition［C］∥ Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition．San Francisco：IEEE，2010：998-1005.
5	PENG S， LIU Y， HUANG Q，et al ．PVNet：pixel-wise voting network for 6DoF pose estimation［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：4561-4570.
6	HE Y， SUN W， HUANG H，et al ．PVN3D：a deep point-wise 3D keypoints voting network for 6DoF pose estimation［C］∥ Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：11632-11641.
7	XIANG Y， SCHMIDT T， NARAYANAN V，et al ．PoseCNN：a convolutional neural network for 6D object pose estimation in cluttered scenes［EB/OL］．（2017-11-01）［2023-04-22］．.
8	WANG C， XU D， ZHU Y，et al ．DenseFusion：6D object pose estimation by iterative dense fusion［C］∥Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：3343-3352.
9	MO N， GAN W， YOKOYA N，et al ．E S6D：a computation efficient and symmetry-aware 6D pose regression framework［C］∥ Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition．New Orleans：IEEE，2022：6718-6727.
10	HAGELSKJÆR F， BUCH A G ．Bridging the reality gap for pose estimation networks using sensor-based domain randomization［C］∥ Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops．Montreal：IEEE，2021：935-944.
11	RAMBACH J， DENG C， PAGANI A，et al ．Learning 6DoF object poses from synthetic single channel images［C］∥ Proceedings of 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct．Munich：IEEE，2018：164-169.
12	HINTERSTOISSER S， PAULY O， HEIBEL H，et al ．An annotation saved is an annotation earned：using fully synthetic training for object detection［C］∥ Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshops．Seoul：IEEE，2019：2787-2796.
13	NOGUES F C， HUIE A， DASGUPTA S ．Object detection using domain randomization and generative adversarial refinement of synthetic images［EB/OL］．（2018-05-30）［2023-05-05］．.
14	DENNINGER M， SUNDERMEYER M， WINKELBAUER D，et al ．BlenderProc［EB/OL］．（2019-10-25）［2023-04-12］．.
15	ZHU J Y， PARK T， ISOLA P，et al ．Unpaired image-to-image translation using cycle-consistent adversarial networks［C］∥ Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：2223-2232.
16	WANG C Y， BOCHKOVSKIY A， LIAO H Y M ．YOLOv7：trainable bag-of-freebies sets new state-of-the-art for real-time object detectors［C］∥ Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Vancouver：IEEE，2023：7464-7475.
17	GONZALEZ A ．Measurement of areas on a sphere using Fibonacci and latitude-longitude lattices［J］．Mathematical Geosciences，2010，42（1）：49-64.
18	HODAŇ T， MATAS J， OBDRŽÁLEK Š ．On evaluation of 6D object pose estimation［C］∥ Proceedings of Computer Vision-ECCV 2016 Workshops．Amsterdam：Springer，2016：606-619.

实验序号	Box_mAP50/%	Mask_mAP50/%
1	74.6	73.5
2	90.2	89.7

数据集	角码数量	连接件数量	铝型材数量
训练集	22 070	10 982	10 855
验证集	2 020	1 206	1 109
测试集	396	86	178

对象类别	个数	抓取次数	抓取成功次数	抓取成功率/%	失败次数
角码	80	106	77	72.6	29
连接件	40	50	38	76.0	12
铝型材	30	35	30	85.7	5

[1]	JI Xiang, WANG Haihong, ZHAI Tiansong, et al.. Global Output Feedback Finite-Time Synchronization Regulation of Robot Manipulators Using Only Position Measurements [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 56-68.
[2]	Liang Peng Li Shao-fa Qin Jiang-wei. Object Recognition Approach Based on Local Type-Consistent k-Means Clustering [J]. Journal of South China University of Technology (Natural Science Edition), 2011, 39(2): 118-124.
[3]	E Jia-qiang Wang Yao-nan Gong Jin-ke Li Chun-sheng Mei Chi. Soft-Sensing Model of Chaos Time Series of Crude Copper Composition in Copper Convertor and Its Application [J]. Journal of South China University of Technology (Natural Science Edition), 2006, 34(1): 52-56.