An Open-World Object Detection Method of Capable of Addressing Label Bias Issues

doi:10.12141/j.issn.1000-565X.240109

Abstract

Abstract:

Open World Object Detection (OWOD) extends the problem of object detection to more complex real-world dynamic scenarios, requiring the system to recognize all known and unknown object categories in the image and possess the capability for incremental learning based on newly introduced knowledge. However, current OWOD methods typically mark regions with high object scores as unknown objects and largely rely on supervision of known objects. Although these methods can detect unknown objects that are similar to known ones, they suffer from a significant label bias problem, where regions dissimilar to known objects are often misclassified as part of the background. To address this issue, this study first proposed an unsupervised region proposal generation method based on a large visual model to enhance the model’s ability to detect unknown objects. Then, considering that the sensitivity of the Region of Interest (ROI) classification stage to new categories during model training can affect the generalization performance of the Region Proposal Network (RPN) in the proposal generation stage, a decoupled joint training method for RPN region proposal generation and ROI classification was introduced to improve the model's capability to resolve label bias problems. Experimental results show that the method proposed in this study has achieved a significant improvement in detecting unknown objects on the MS-COCO dataset, with the unknown category recall rate exceeding that of the previous SOTA methods by more than twice, reaching 52.1%, while maintaining competitiveness in detecting known object categories. In terms of inference speed, the model, constructed using pure convolutional neural networks rather than dense attention mechanisms, achieves a frame rate 8.18 f/s higher than that of deformable DETR-based methods.

Key words: unsupervision, open world, incrementally learn, object detection

CLC Number:

TP391

HUANG Yangyang, XU Yong, XI Xing, LUO Ronghua. An Open-World Object Detection Method of Capable of Addressing Label Bias Issues[J]. Journal of South China University of Technology(Natural Science Edition), 2025, 53(3): 12-19.

Figures/Tables 5

Fig.1

Table 1

Table 2

Table 3

Fig.2

References 26

1	REN S， HE K， GIRSHICK R，et al ． Faster R-CNN：towards real-time object detection with region proposal networks ［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（6）：1137-1149.
2	REDMON J， DIVVALA S， GIRSHICK R，et al ．You only look once：unified，real-time object detection ［C］∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：779-788.
3	LIN T Y， GOYAL P， GIRSHICK R，et al ．Focal loss for dense object detection ［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，42（2）：318-327.
4	ZHU X， SU W， LU L，et al ． Deformable DETR：deformable transformers for end-to-end object detection ［C］∥ Proceedings of the 9th International Conference on Learning Representations．Vienna：OpenReview.net，2021：1-16.
5	DHAMIJA A， GÜNTHER M， VENTURA J，et al ．The overlooked elephant of object detection：open set ［C］∥ Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision．Snowmass：IEEE，2020：1010-1019.
6	JOSEPH K J， KHAN S， KHAN F S，et al ．Towards open world object detection ［C］∥ Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Nashville：IEEE，2021：5826-5836.
7	GUPTA A， NARAYAN S， JOSEPH K J，et al ．OW-DETR：open-world detection transformer［C］∥ Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition．New Orleans：IEEE，2022：9225-9234.
8	ZOHAR O， WANG K C， YEUNG S ．PROB：probabilistic objectness for open world object detection ［C］∥ Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Vancouver：IEEE，2023：11444-11453.
9	MA S， WANG Y， WEI Y，et al ．CAT：localization and identification cascade detection transformer for open-world object detection ［C］∥ Proceedings of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition．Vancouver：IEEE，2023：19681-19690.
10	DONG N， ZHANG Y， DING M，et al ．Open world DETR：transformer based open world object detection ［EB/OL］．（2022-12-06）［2024-03-05］．.
11	WANG X， YU Z， DE MELLO S，et al ．FreeSOLO：learning to segment objects without annotations ［C］∥ Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans：IEEE，2022：4156-4166.
12	BAR A， WANG X， KANTOROV V，et al ．DETReg：unsupervised pretraining with region priors for object detection ［C］∥ Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition．New Orleans：IEEE，2022：14585-14595.
13	KIRILLOV A， MINTUN E， RAVI N，et al ．Segment anything ［C］∥ Proceedings of 2023 IEEE/CVF International Conference on Computer Vision．Paris：IEEE，2023：3992-4003.
14	ZHOU Y ．Rethinking reconstruction autoencoder-based out-of-distribution detection ［C］∥ Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition．New Orleans：IEEE，2022：7369-7377.
15	JIANG W， GE Y， CHENG H，et al ． READ：aggregating reconstruction error into out-of-distribution detection ［C］∥ Proceedings of the 37th AAAI Conference on Artificial Intelligence．Washington D C：AAAI，2023：14910-14918.
16	OSADA G， TAKAHASHI T， AHSAN B，et al ．Out-of-distribution detection with reconstruction error and typicality-based penalty ［C］∥ Proceedings of 2023 IEEE/CVF Winter Conference on Applications of Computer Vision．Waikoloa：IEEE，2023：5540-5552.
17	FANG R H， PANG G S， ZHOU L，et al ．Unsupervised recognition of unknown objects for open-world object detection ［EB/OL］．（2023-08-31）［2024-03-05］．.
18	SHMELKOV K， SCHMID C， ALAHARI K ．Incremental learning of object detectors without catastrophic forgetting［C］∥ Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：3420-3429.
19	HAO Y， FU Y， JIANG Y，et al ．An end-to-end architecture for class-incremental object detection with knowledge distillation ［C］∥ Proceedings of 2019 IEEE International Conference on Multimedia and Expo．Shanghai：IEEE，2019：1-6.
20	YANG B， DENG X， SHI H，et al ．Continual object detection via prototypical task correlation guided gating mechanism［C］∥ Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition．New Orleans：IEEE，2022：9245-9254.
21	HE K， GKIOXARI G， DOLLÁR P，et al ．Mask R-CNN ［C］∥ Proceedings of 2017 IEEE International Conference on Computer Vision．Venice：IEEE，2017：2980-2988.
22	WEI F， GAO Y， WU Z，et al ．Aligning pretraining for detection via object-level contrastive learning［C］∥ Proceedings of the 35th International Conference on Neural Information Processing Systems．Red Hook：Curran Associates Inc.， 2021：22682-22694.
23	LI Z， HOIEM D ．Learning without forgetting［C］∥Proceedings of the 14th European Conference on Computer Vision．Amsterdam：Springer，2016：614-629.
24	DHAR P， SINGH R V， PENG K C，et al ．Learning without memorizing［C］∥ Proceedings of 2019 IEEE/CVF Conference on Computer Visionand Pattern Recognition．Long Beach：IEEE，2019：5133-5141.
25	HE K， ZHANG X， REN S，et al ．Deep residual learning for image recognition ［C］∥ Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：770-778.
26	UIJLINGS J R， GEVERS T， SMEULDERS W A ．Selective search for object recognition［J］．International Journal of Computer Vision，2013，104（2）：154-171.

模型	召回率/%				mAP_pk/%				mAP_ck/%				mAP_bo/%
模型	T₁	T₂	T₃	T₄	T₁	T₂	T₃	T₄	T₁	T₂	T₃	T₄	T₁	T₂	T₃	T₄
ORE-EBUI	1.5	3.9	3.6			61.0	43.1	33.6	71.4	30.9	32.2	26.3		45.6	39.5	31.8
OW-DETR	5.7	6.2	6.9			65.0	46.7	38.2	73.1	29.0	25.7	28.1		46.0	39.7	33.1
PROB	17.6	22.3	24.8			66.3	51.2	42.6	73.5	36.0	30.4	31.7		50.4	42.0	39.9
CAT	24.0	23.0	24.6			67.6	50.2	45.4	74.2	35.5	32.6	35.1		50.7	45.0	42.8
文中模型	52.1	50.4	45.7			67.1	48.3	43.5	73.3	36.5	33.9	33.6		51.6	43.5	40.9

无监督区域提议生成	召回率/%	mAP_ck/%
基线方法	33.4	73.2
SS	34.2	73.7
DETReg	37.1	73.4
FreeSOLO	37.9	74.0
SAM-URGP	52.1	73.3

序号	SAM-URGP	ORWM	DTRR	召回率/%	mAP_ck/%
1	√	×	√	40.3	68.4
2	√	√	√	52.1	73.3
3	√	×	×	45.2	69.7
4	√	√	×	50.9	73.5

[1]	LUO Yutao, MAO Haojie. Single-Stage Object Detection Algorithm with Enhanced Pillar Feature Encoding [J]. Journal of South China University of Technology(Natural Science Edition), 2025, 53(3): 1-11.
[2]	YUE Yongheng, LEI Wenpeng. Foggy Road Environment Perception Algorithm Based on an Improved CycleGAN and YOLOv8 [J]. Journal of South China University of Technology(Natural Science Edition), 2025, 53(2): 48-57.
[3]	YUE Yongheng, NING Ruihou. Intelligent Vehicle Object Detection Algorithm Based on Lightweight CenterNet [J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(8): 45-55.
[4]	GUO Enqiang, FU Xinsha. Dropped Object Detection Method Based on Feature Similarity Learning [J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(6): 30-41.
[5]	DU Qiliang, XIANG Zhaoyi, TIAN Lianfang. Real Time Statistics Method of Escalator Passenger Flow for Embedded Devices [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(6): 60-70.
[6]	YI Qingming, LÜ Renyi, SHI Min, et al. Lightweight Object Detection Combined with Multi-Scale Dilated-Convolution and Multi-Scale Deconvolution [J]. Journal of South China University of Technology(Natural Science Edition), 2022, 50(12): 41-48.
[7]	TAN Guang LI Changhao ZHAN Zhaohuan. Adaptive Scheduling Algorithm for Object Detection and Tracking Based on Device-Cloud Collaboration [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(7): 86-93.
[8]	LI Bo RAO Haobo. Salient Object Detection Based on Feature Enhancement in Complex Scene [J]. Journal of South China University of Technology (Natural Science Edition), 2021, 49(11): 135-144.
[9]	LIU Weiming, WEN Junrui, ZHENG Zhongxing, et al. DifferentNet: Neural Network for Foreign Objects Foreground Detection in Metro [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(10): 11-21,40.
[10]	WU Qiuxia, LI Lingmin. 3D Object Detection Based on Point Cloud Bird's Eye View Remapping [J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(1): 39-46.
[11]	LIU Yujie CAO Xianzhi LI Zongmin LI Hua. Video Object Detection Based on Correlation Feature and Convolutional Neural Network [J]. Journal of South China University of Technology (Natural Science Edition), 2018, 46(12): 26-33.
[12]	Fu Xin-sha Wang Xiang-bo Li Hai-feng Meng Qing-xin. Detection Algorithm of Expressway Moving Objects Based on Background Subtraction [J]. Journal of South China University of Technology (Natural Science Edition), 2015, 43(4): 1-6.
[13]	Guo Ling Li Zhen-zhen Du Ming-hui. Moving Target Tracking Based on Monocular Sequence Images [J]. Journal of South China University of Technology(Natural Science Edition), 2012, 40(3): 94-99.
[14]	Luo Rong-hua Min Hua-qing. Hybrid Conditional Random Field for Multi-Object Tracking of Mobile Robot [J]. Journal of South China University of Technology (Natural Science Edition), 2011, 39(5): 61-67.