Cross-Domain Pressure Footprint Images Retrieval Based on Mutual Information Disentangled Representations

doi:10.12141/j.issn.1000-565X.220572

Abstract

Abstract:

As one of human biometric features, footprint is of great significance in the field of biometric identification. However, the pressure footprint images of different shoe types for the same person have significant differences in the footprint contour features, leading to large intra-class differences. For cross-domain retrieval of pressure footprint images, this paper proposed a cross-domain pressure footprint images retrieval method based on mutual information disentangled representations. Firstly, a multi-domain pressure footprint dataset containing 200 people’s footprint images was constructed and the characteristics of cross-domain pressure footprint images were analyzed from qualitative and quantitative perspectives. Secondly, two independent encoders were used to construct an image disentanglement module, which disentangles the pressure footprint images into a domain-specific representation and a domain-shared representation, and ensures that the domain-specific representation contains more domain-related information through domain classification. Then, the distance between the domain-specific representation and the domain-shared representation was enlarged by minimizing mutual information loss. At the same time, in order to avoid the loss of information in the disentangled process, the original pressure footprint image was reconstructed based on the domain-specific representation and the domain-shared representation. Finally, the deep convolution features of the domain-shared representation were further extracted by feature extraction module and the cross-domain pressure footprint images retrieval was realized through the metric module which calculates the correlation degree between different features. The results of comparison and ablation experiments show that the disentanglement module of this method is effective and performs well on multi-domain pressure footprint dataset. The retrieval accuracy of the first query result reached 79.83%, and the average accuracy reached 65.48%.

Key words: image retrieval, cross-domain pressure footprint, disentangled representation, domain-shared representation, mutual information

CLC Number:

TP391

ZHANG Yan, XU Changkang, CAO Liqing, et al. Cross-Domain Pressure Footprint Images Retrieval Based on Mutual Information Disentangled Representations[J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(5): 78-85.

Figures/Tables 11

Fig.1

Fig.2

Table 1

Fig.3

Table 2

Table 3

Table 4

Fig.4

Table 5

Fig.5

Table 6

References 19

1	金益锋，白艳平，刘寰．全国16个省份足迹自动识别系统应用情况分析［J］．刑事技术，2017，42（6）：504-507.
	JIN Yifeng， BAI Yanping， LIU Huan ．Application analysis on shoeprint automatic identification system from China’s 16 provinces［J］．Forensic Science and Technology，2017，42（6）：504-507.
2	史力民，马建平．足迹学［M］．北京：中国人民公安大学出版社，2014：1-12.
3	KANCHAN T， MENEZES R G， MOUDGIL R，et al ．Stature estimation from foot dimensions［J］．Forensic Science International，2008，179（2/3）：241.e1-241.e5.
4	KEATSAMARN T， PINTAVIROOJ C ．Footprint identification using deep learning［C］∥ Proceedings of 2018 the 11th Biomedical Engineering International Conference．Chiang Mai：IEEE，2018：1-4.
5	陈杨，曾诚，程成，等．一种基于CNN的足迹图像检索与匹配方法［J］．南京师范大学学报（工程技术版），2018，18（3）：39-45.
	CHEN Yang， ZENG Cheng， CHENG Cheng，et al ．A CNN-based approach to footprint image retrieval and matching［J］．Journal of Nanjing Normal University （Engineering and Technology Edition），2018，18（3）：39-45.
6	鲍文霞，瞿金杰，王年，等．基于空间聚合加权卷积神经网络的力触觉足迹识别［J］．东南大学学报（自然科学版），2020，50（5）：959-964.
	BAO Wenxia， QU Jinjie， WANG Nian，et al ．Force-tactile footprint recognition based on spatial aggregation weighted convolutional neural network［J］．Journal of Southeast University （Natural Science Edition），2020，50（5）：959-964.
7	张艳，吴洛天，王年，等．基于多模块关系网络的2D足迹分类［J］．华南理工大学学报（自然科学版），2021，49（6）：66-76.
	ZHANG Yan， WU Luotian， WANG Nian，et al ．2D footprint classification based on multiple-module relation network［J］．Journal of South China University of Technology （Natural Science Edition），2021，49（6）：66-76.
8	鲍文霞，茅丽丽，王年，等．基于注意力双分支网络的跨模态足迹检索［J］．东南大学学报（自然科学版），2021，51（5）：914-922.
	BAO Wenxia， MAO Lili， WANG Nian，et al ．Cross-modal footprint retrieval based on the two-branch CNN with attention［J］．Journal of Southeast University （Natural Science Edition），2021，51（5）：914-922.
9	李浩然，周小平，王佳．跨域图像检索综述［J］．计算机工程与应用，2022，58（15）：18-36.
	LI Haoran， ZHOU Xiaoping， WANG Jia ．Review of cross-domain image retrieval［J］．Computer Engineering and Applications，2022，58（15）：18-36.
10	LIU F C， GAO C Q， SUN Y Q，et al ．Infrared and visible cross-modal image retrieval through shared features［J］．IEEE Transactions on Circuits and Systems for Video Technology，2021，31（11）：4485-4496.
11	PAUL S， DUTTA T， BISWAS S ．Universal cross-domain retrieval：generalizing across classes and domains［C］∥ Proceedings of 2021 IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：12036-12044.
12	YU Q， SONG J， SONG Y Z，et al ．Fine-grained instance-level sketch-based image retrieval ［J］．International Journal of Computer Vision，2021，129（2）：484-500.
13	CHEN Y D， ZHANG Z L， WANG Y F，et al ．AE-Net：fine-grained sketch-based image retrieval via attention-enhanced network［J］．Pattern Recognition，2022，122：108291/1-15.
14	LEE H Y， TSENG H Y， MAO Q，et al ．DRIT++：diverse image-to-image translation via disentangled representations［J］．International Journal of Computer Vision，2020，128：2402-2417.
15	WU A， HAN Y H， ZHU L，et al ．Instance-invariant domain adaptive object detection via progressive disentanglement ［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2022，44（8）：4178-4193.
16	BENGIO Y， COURVILLE A， VINCENT P ．Representation learning：a review and new perspectives［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2013，35（8）：1798-1828.
17	BELGHAZI M I， BARATIN A， RAJESWAR S，et al ．Mutual information neural estimation［C］∥ Proceedings of the 35th International Conference on Machine Learning．Stockholm：IMLS，2018：864-873.
18	PENG X， HUANG Z， ZHU Y，et al ．Federated adversarial domain adaptation［C］∥ Proceedings of the Eighth International Conference on Learning Representations．Ethiopia：ICLR，2020：1-19.
19	HWANG H J， KIM G H， HONG S，et al ．Variational interaction information maximization for cross-domain disentanglement［C］∥ Proceedings of the 34th Conference on Neural Information Processing Systems．Vancouver：NIPS Foundation，2020：1-26.

鞋型	类别总数		图像总数
鞋型	训练集	测试集	训练集	测试集
布鞋	140	60	5 600	600
运动鞋	140	60	5 600	600

图像	SSIM
图像	a₁	a₂	a₃	a₄	均值
均值	0.891 3	0.890 3	0.885 3	0.886 5	0.888 4
a₁		0.897 4	0.884 9	0.891 7	0.891 3
a₂	0.897 4		0.888 3	0.885 1	0.890 3
a₃	0.884 9	0.888 3		0.882 7	0.885 3
a₄	0.891 7	0.885 1	0.882 7		0.886 5

图像	SSIM
图像	b₁	b₂	b₃	b₄	均值
均值	0.871 8	0.873 3	0.872 3	0.871 6	0.872 2
a₁	0.872 3	0.873 6	0.871 0	0.870 5	0.871 9
a₂	0.871 2	0.869 5	0.868 5	0.872 2	0.870 3
a₃	0.867 1	0.872 3	0.878 6	0.874 6	0.873 1
a₄	0.876 5	0.877 6	0.871 0	0.869 2	0.873 6

方法	Rank1	Rank5	Rank10	mAP
ResNet152	67.12	86.61	92.37	57.53
IIAE	74.28	89.93	94.79	61.76
MI-DR（仅源域）	9.32	16.44	22.37	9.09
MI-DR（DSpR）	8.48	25.09	37.29	6.50
MI-DR（DShR）	79.83	93.22	96.78	65.48

骨干网络	Rank1	Rank5	Rank10	mAP
AlexNet	62.93	83.32	89.41	50.56
VGG19	66.23	86.89	90.95	53.14
ResNet18	67.80	87.29	92.54	53.84
ResNet34	74.58	89.49	93.56	59.60
ResNet50	74.58	92.54	96.10	60.32
ResNet101	71.53	88.14	94.24	60.65
ResNet152	79.83	93.22	96.78	65.48