An Autonomous Extrinsic Calibration Method for 4D Millimeter-Wave Radar Point Clouds and Visual Images

doi:10.12141/j.issn.1000-565X.240172

Abstract

Abstract:

With the rapid development of autonomous driving technology, the demand for multi-sensor fusion in environmental perception systems is increasing. Four-dimensional (4D) millimeter-wave radar has become one of the critical sensors in autonomous driving due to its stable performance under complex weather and lighting conditions. Although 4D millimeter-wave radar improves object detection accuracy by adding elevation angle information and increasing point cloud density, its sparse point clouds and noise issues limit its independent application. Therefore, the fusion of 4D millimeter-wave radar with vision sensors has become key to enhancing perception accuracy in autonomous driving. However, traditional extrinsic calibration methods rely on cumbersome manual operations, making it challenging to meet the requirements for efficient automated calibration. To address this issue, this study proposed an automated extrinsic calibration method for 4D millimeter-wave radar and visual images based on a calibration board. The method first designs a calibration board with ChArUco markers, red circular rings, and corner reflectors, and then automatically extracts image coordinates and radar point cloud coordinates of the calibration points using a circle detection algorithm and a corner reflector detection algorithm. Furthermore, a method for calibration data acquisition and validation was proposed using simulations in 3D Max and Unity. Finally, the performance of direct linear transformation (DLT) and extrinsic calibration (EC) methods is compared through experiments to evaluate calibration accuracy. Experimental results indicate that the designed calibration board and automated calibration algorithm effectively reduce manual operations, and the EC method demonstrates higher calibration stability and accuracy when more calibration points are involved.

Key words: multi-sensor calibration, 4D millimeter-wave radar, vision, autonomous driving, extrinsic calibration

CLC Number:

TP212

BI Xin, WENG Caien, WANG Yu, HU Zaigang. An Autonomous Extrinsic Calibration Method for 4D Millimeter-Wave Radar Point Clouds and Visual Images[J]. Journal of South China University of Technology(Natural Science Edition), 2025, 53(1): 74-83.

Figures/Tables 12

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Table 1

Fig.10

Fig.11

References 26

1	YAO S， GUAN R， HUANG X，et al ．Radar-camera fusion for object detection and semantic segmentation in autonomous driving：a comprehensive review［J］．IEEE Transactions on Intelligent Vehicles，2024，9（1）：2094-2128．
2	ZHOU Y， LIU L， ZHAO H，et al ．Towards deep radar perception for autonomous driving：datasets，methods，and challenges［J］．Sensors，2022，22（11）：4208/1-45．
3	HAN Z， WANG J， XU Z，et al ．4D millimeter-wave radar in autonomous driving：a survey［EB/OL］．（2023-06-07）［2024-04-01］．．
4	林永杰，陈宁，卢凯．基于毫米波雷达点云的路口车辆轨迹跟踪方法［J］．华南理工大学学报（自然科学版），2023，51（10）：110-125．
	LIN Yongjie， CHEN Ning， LU Kai ．Vehicle trajectory tracking at intersections based on millimeter wave radar point cloud［J］．Journal of South China University of Technology（Natural Science Edition），2023，51（10）：110-125．
5	SINGH A ．Vision-radar fusion for robotics BEV detections：a survey［C］∥ Proceedings of 2023 IEEE Intelligent Vehicles Symposium．Anchorage：IEEE，2023：10186647/1-7．
6	XIONG W， LIU J， HUANG T，et al ．LXL：LiDAR excluded lean 3D object detection with 4D imaging radar and camera fusion［J］．IEEE Transactions on Intelligent Vehicles，2024，9（1）：79-92．
7	ZHENG L， LI S， TAN B，et al ．RCFusion：fusing 4-D radar and camera with bird’s-eye view features for 3D object detection［J］．IEEE Transactions on Instrumentation and Measurement，2023，72：8503814/1-14．
8	DOMHOF J， KOOIJ J F P， GAVRILA D M ．An extrinsic calibration tool for radar，camera and lidar［C］∥ Proceedings of 2019 International Conference on Robotics and Automation．Montreal：IEEE，2019：8107-8113．
9	ZHANG J， ZHANG S， PENG G，et al ．3DRadar2ThermalCalib：accurate extrinsic calibration between a 3D mmWave radar and a thermal camera using a spherical-trihedral［C］∥ Proceedings of 2022 IEEE the 25th International Conference on Intelligent Transportation Systems．Macau：IEEE，2022：2744-2749．
10	KIM D， KIM S ．Extrinsic parameter calibration of 2D radar-camera using point matching and generative optimization［C］∥ Proceedings of 2019 the 19th International Conference on Control，Automation and Systems．Jeju：IEEE，2019：99-103．
11	LIU M， LI D， LI Q，et al ．An online intelligent method to calibrate radar and camera sensors for data fusing［J］．Journal of Physics：Conference Series，2020，1631：012183/1-9．
12	SHEENY M， DE PELLEGRIN E， MUKHERJEE S，et al ．RADIATE：a radar dataset for automotive perception in bad weather［C］∥ Proceedings of 2021 IEEE International Conference on Robotics and Automation．Xi’an：IEEE，2021：5617-5623．
13	SCHÖLLER C， SCHNERRLER M， KRÄMMER A，et al ．Targetless rotational auto-calibration of radar and camera for intelligent transportation systems［C］∥ Proceedings of 2019 IEEE Intelligent Transportation Systems Conference．Auckland：IEEE，2019：3934-3941．
14	WANG X Y， WANG X S， ZHOU Z Q ．A high-accuracy calibration method for fusion systems of millimeter-wave radar and camera［J］．Measurement Science and Technology，2022，34（1）：015103/1-14．
15	CUI H， WU J， ZHANG J，et al ．3D detection and tracking for on-road vehicles with a monovision camera and dual low-cost 4D mmWave radars［C］∥ Procee-dings of 2021 IEEE International Intelligent Transportation Systems Conference．Indianapolis：IEEE，2021：2931-2937．
16	OH J， KIM K S， PARK M，et al ．A comparative study on camera-radar calibration methods［C］∥ Proceedings of 2018 the 15th International Conference on Control，Automation，Robotics and Vision．Singapore：IEEE，2018：1057-1062．
17	CHENG L， SENGUPTA A， CAO S ．3D radar and camera co-calibration：a flexible and accurate method for target-based extrinsic calibration［C］∥ Proceedings of 2023 IEEE Radar Conference．San Antonio：IEEE，2023：10149669/1-6．
18	WISE E， PERŠIC J， GREBE C，et al ．A continuous-time approach for 3D radar-to-camera extrinsic calibration［C］∥ Proceedings of 2021 IEEE International Conference on Robotics and Automation．Xi’an：IEEE，2021：13164-13170．
19	WIES E， CHENG Q， KELLY J ．Spatiotemporal calibration of 3-D millimetre-wavelength radar-camera pairs［J］．IEEE Transactions on Robotics，2023，39（6）：4552-4566．
20	GAO D， DUAN J， YANG X，et al ．A method of spatial calibration for camera and radar［C］∥ Proceedings of 2010 the 8th World Congress on Intelligent Control and Automation．Jinan：IEEE，2010：6211-6215．
21	WANG T， ZENG N， XIN J，et al ．Integrating millimeter wave radar with a monocular vision sensor for on-road obstacle detection applications［J］．Sensors，2011，11（9）：8992-9008．
22	ZHOU Y， DONG Y， HOU F，et al ．Review on millimeter-wave radar and camera fusion technology ［J］．Sustainability，2022，14（9）：5114/1-32．
23	JI Z， PROKHOROV D ．Radar-vision fusion for object classification［C］∥ Proceedings of 2008 the 11th International Conference on Information Fusion．Cologne：IEEE，2008：1-7．
24	Inc GitHub ．OpenCV：open source computer vision library［EB/OL］．（2022-12-28）［2024-04-01］．．
25	LEE C L， HSUEH Y H， WANG C C，et al ．Extrinsic and temporal calibration of automotive radar and 3D LiDAR［C］∥ Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems．Las Vegas：IEEE，2020：9976-9983．
26	AGRAWAL S， BHANDERI S， DOYCHEVA K，et al ．Static multitarget-based autocalibration of RGB cameras，3-D rada，and 3-D lidar sensors［J］．IEEE Sensors Journal，2023，23（18）：21493-21505．

N	平均重投影误差
N	DLT方法	EC方法
6	0.00	1.90
8	1.19	2.11
10	0.42	1.71
12	0.73	1.84
14	47.06	1.69
18	40.74	1.60
24	35.79	2.05

[1]	DU Qiliang, WANG Yimin, TIAN Lianfang. Attention Module Based on Feature Similarity and Feature Normalization [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(7): 62-71.
[2]	HE Yongming, XING Wanyu, WEI Kun, et al. Lane-Changing Trajectory Planning Strategy for Autonomous Vehicles on Superhighways [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(4): 104-113.
[3]	CHEN Qiong, FENG Yuan, LI Zhiqun, et al. Semantic-Visual Consistency Constraint Network for Zero-Shot Image Semantic Segmentation [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 41-50.
[4]	LUO Yutao, XUE Zhicheng. Multi-Task Assisted Driving Policy Learning Method for Autonomous Driving [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(10): 31-40.
[5]	WANG Shiyong, QIAN Guokang, LI Di, et al.. Real-Time Template Matching Method for Edge Features [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 1-10.
[6]	ZHUANG Ling, HUANG Kai. RIS-IM System Based on Nonorthogonal Multiple Access [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(11): 28-34.
[7]	WANG Gao, CHEN Xiaohong, LIU Ning, et al. A Robot Grasping Policy Based on Viewpoint Selection Experience Enhancement Algorithm [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(9): 126-137.
[8]	LI Beibei, WEI Lingxing, LIU Xiumei, et al. Vibration Testing of Poppet Valve Based on Virtual Binocular Vision [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(10): 106-113.
[9]	ZHANG Wendong. SHI Gang, TIAN Shengwei, et al. Social Relationship-Based Task Distribution Mechanism of Crowdsensing [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(6): 49-55.
[10]	GENG Qinghua, LIU Weiming. A Fast and Stable Algebraic Solution to Perspective-Three-Point Problem [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(1): 58-64,73.
[11]	HONG Bo, LI Peng, WU Hongbao, et al. Application of Interpolation Loop Surface Subdivision Algorithm in Free Surface Reconstruction of Surfacing [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(8): 139-144.
[12]	TIAN Sheng, XU Kai, ZHU Zekun, et al. Day-to-Day Evolution Model of Road-Network Mixed Traffic Flow in Autonomous Driving Environment [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(4): 123-131.
[13]	XIA Yimin, MA Jiesong, ZHANG Yazhou, et al. Bolting Jumbo Boom Positioning Based on Compliance Error Detection [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(3): 83-90.
[14]	ZHENG Sifan, WANG Weixing, HE Zhanhua, et al. Research on Swing Amplitude Detection of Automobile Wiper with Two Granularity Optical Flow Manifold Learning#br# [J]. Journal of South China University of Technology (Natural Science Edition), 2020, 48(1): 123-132.
[15]	ZHAI Jingmei HUANG Jinzhou LIU Kun. Detection Method Based on Edge Feature with Binocular Vision for the Working Environment of Ｒobot [J]. Journal of South China University of Technology (Natural Science Edition), 2018, 46(3): 1-7.