Attention Module Based on Feature Similarity and Feature Normalization

doi:10.12141/j.issn.1000-565X.230313

Abstract

Abstract:

In recent years, attention mechanisms have achieved great success in the fields of image classification, object detection and semantic segmentation. However, most existing attention mechanisms can only achieve feature fusion in channel or spatial dimensions, which greatly limits the flexibility of attention mechanisms to change in channel and spatial dimensions and cannot fully utilize feature information. To address this issue, this paper proposes a convolutional neural network attention module based on feature similarity and feature normalization (FSNAM), which can utilize the characteristic information of both channel domain and spatial domain. FSNAM consists of a feature similarity module (FSM) and a feature normalization module (FNM). FSM generates a two-dimension feature similarity weight map using the channel feature information and local spatial feature information of the input feature map, while FNM generates a three-dimension feature normalization weight map using the global spatial feature information of the input feature map. The weight maps generated by FSM and FNM are fused to generate a three-dimension attention weight map to achieve the fusion of channel feature information and spatial feature information. Moreover, to demonstrate the feasibility and effectiveness of FSNAM, ablation experiments are conducted. The results show that, for image classification tasks, FSNAM significantly outperforms other mainstream attention modules in improving the performance of the classification network on CIFAR dataset; for object detection tasks, the object detection network using FSNAM improves the detection accuracy of small and medium-sized objects in VOC dataset by 3.9 and 1.2 points of percentage, respectively; and, for semantic segmentation tasks, FSNAM can significantly improve the performance of HRNet model, and helps to achieve an average pixel accuracy increase of the model on SBD dataset of 0.58 points of percentage.

Key words: convolutional neural network, computer vision, feature similarity, feature normalization, attention module

CLC Number:

TP391.4

DU Qiliang, WANG Yimin, TIAN Lianfang. Attention Module Based on Feature Similarity and Feature Normalization[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(7): 62-71.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Table 1

Table 2

Table 3

Table 4

Fig.4

Table 5

Table 6

References 29

1	DENG J， DONG W， SOCHER R，et al ．Imagenet：a large-scale hierarchical image database［C］∥Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition．Miami：IEEE，2009：248-255．
2	LIN T Y， MAIRE M， BELONGIE S，et al ．Microsoft COCO：common objects in context［C］∥Proceedings of the 13th European Conference on Computer Vision．Zurich：Springer，2014：740-755．
3	HE K， ZHANG X， REN S，et al ．Deep residual learning for image recognition［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Las Vegas：IEEE，2016：770-778．
4	HUANG G， LIU Z， VAN DER MAATEN L，et al ．Densely connected convolutional networks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Honolulu：IEEE，2017：4700-4708．
5	HU J， SHEN L， SUN G ．Squeeze-and-excitation networks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：7132-7141．
6	DAI T， CAI J， ZHANG Y，et al ．Second-order attention network for single image super-resolution［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：11065-11074．
7	ZHAO H， KONG X， HE J，et al ．Efficient image super-resolution using pixel attention［C］∥Proceedings of the European Conference on Computer Vision．Glasgow：Springer，2020：56-72．
8	MNIH V， HEESS N， GRAVES A ．Recurrent models of visual attention［J］．Advances in Neural Information Processing Systems，2014，2（12）：2204-2212．
9	BA J， MNIH V， KAVUKCUOGLU K ．Multiple object recognition with visual attention［EB/OL］．（2015-04-23）［2023-05-10］．．
10	XU K， BA J， KIROS R，et al ．Show，attend and tell：neural image caption generation with visual attention［C］∥Proceedings of the International Conference on Machine Learning．Lille：PMLR，2015：2048-2057．
11	GREGOR K， DANIHELKA I， GRAVES A，et al ．DRAW：a recurrent neural network for image generation［C］∥Proceedings of the International Conference on Machine Learning．Lille：PMLR，2015：1462-1471．
12	WOO S， PARK J， LEE J Y，et al ．CBAM：convolutional block attention module［C］∥Proceedings of the European Conference on Computer Vision．Munich：Springer，2018：3-19．
13	WANG X， GIRSHICK R， GUPTA A，et al ．Non-local neural networks［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition．Salt Lake City：IEEE，2018：7794-7803．
14	WANG Q， WU B， ZHU P，et al ．ECA-Net：efficient channel attention for deep convolutional neural networks［C］∥Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition．Seatle：IEEE，2020：11534-11542．
15	PARK J，WOO S， LEE J Y，et al ．BAM：bottleneck attention module［EB/OL］．（2018-07-18）［2023-05-10］．．
16	VASWANI A， SHAZEER N， PARMAR N，et al ．Attention is all you need［J］．Advances in Neural Information Processing Systems，2017，31（17）：6000-6010．
17	LIU Z， LIN Y， CAO Y，et al ．Swin transformer：hie-rarchical vision transformer using shifted windows［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：10012-10022．
18	IOFFE S， SZEGEDY C ．Batch normalization：acce-lerating deep network training by reducing internal covariate shift［C］∥Proceedings of the International Conference on Machine Learning．Lille：PMLR，2015：448-456．
19	LI X， SUN W， WU T ．Attentive normalization［C］∥Proceedings of the European Conference on Computer Vision．Glasgow：Springer，2020：70-87．
20	YAO M， ZHAO G， ZHANG H，et al ．Attention spiking neural networks［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2023，45（8）：9393-9410．
21	YANG L， ZHANG R Y， LI L，et al ．SimAM：a simple，parameter-free attention module for convolutional neural networks［C］∥Proceedings of the International Conference on Machine Learning．Graz：PMLR，2021：11863-11874．
22	WEBB B S， DHRUV N T， SOLOMON S G，et al ．Early and late mechanisms of surround suppression in striate cortex of macaque［J］．Journal of Neuroscience，2005，25（50）：11666-11675．
23	TAN S， ZHANG L， SHU X，et al ．A feature-wise attention module based on the difference with surrounding features for convolutional neural networks［J］．Frontiers of Computer Science，2023，17（6）：338-348．
24	HE K， ZHANG X， REN S，et al ．Identity mappings in deep residual networks［C］∥Proceedings of the 14th European Conference on Computer Vision．Amsterdam：Springer，2016：630-645．
25	EVERINGHAM M， ESLAMI S M A， VAN GOOL L，et al ．The pascal visual object classes challenge：a retrospective［J］．International Journal of Computer Vision，2015，111（1）：98-136．
26	BOCHKOVSKIY A， WANG C Y， LIAO H Y M ．Yolov4：optimal speed and accuracy of object detection［EB/OL］．（2020-04-23）［2023-05-10］．．
27	HARIHARAN B， ARBELAEZ P， BOURDEV L，et al ．Semantic contours from inverse detectors［C］∥ Proceedings of the 2011 International Conference on Computer Vision．Barcelona：IEEE，2011：991-998．
28	LOSHCHILOV I， HUTTER F ．SGDR：stochastic gradient descent with warm restarts［EB/OL］．（2017-03-03）［2023-05-10］．．
29	WANG J， SUN K， CHENG T，et al ．Deep high-resolution representation learning for visual recognition［J］．IEEE Transactions on Pattern Analysis and Machine Intelligence，2021，43（10）：3349-3364．

骨干网络	添加不同注意力模块后的Top-1精度/%
骨干网络	基础网络	SE^［5］	CBAM^［12］	ECA^［14］	SimAM^［21］	文中模块
ResNet-18^［3］	92.41	92.63	92.67	92.54	92.72	92.86
PreResNet-18^［24］	92.37	92.49	92.58	92.72	92.37	92.69
ResNet-34^［3］	93.10	93.42	93.53	93.62	93.53	93.65
PreResNet-34^［24］	93.07	93.34	93.49	93.33	93.28	93.58
ResNet-50^［3］	93.77	93.70	93.78	93.91	93.88	94.02
PreResNet-50^［24］	94.10	94.05	94.13	94.49	94.29	94.58

骨干网络	添加不同注意力模块后的Top-1精度/%
骨干网络	基础网络	SE^［5］	CBAM^［12］	ECA^［14］	SimAM^［21］	文中模块
ResNet-18^［3］	68.87	69.68	69.69	69.03	69.99	70.21
PreResNet-18^［24］	68.69	69.21	69.32	69.91	69.54	69.59
ResNet-34^［3］	71.14	71.48	71.56	71.39	71.34	71.85
PreResNet-34^［24］	70.65	71.12	70.72	71.12	71.08	71.15
ResNet-50^［3］	72.80	73.30	73.28	73.48	73.19	73.61
PreResNet-50^［24］	73.10	74.16	73.94	73.63	74.13	74.01

添加的注意力模块	参数量/10⁶		浮点运算量/10⁶		推理速度/（帧·s^-1）
添加的注意力模块	ResNet-18^［3］	ResNet-50^［3］	ResNet-18^［3］	ResNet-50^［3］	ResNet-18^［3］	ResNet-50^［3］
基础网络	11.181 6	23.528 5	37.220 4	84.342 8	439	210
SE^［5］	11.268 7	26.043 5	37.324 7	86.985 5	310	139
CBAM^［12］	11.271 5	26.061 1	37.411 8	89.401 5	182	77
ECA^［14］	11.181 7	23.528 6	37.243 4	84.528 6	297	133
SimAM^［21］	11.181 6	23.528 5	37.220 4	84.342 8	288	131
文中模块	11.181 7	23.528 6	37.228 0	84.399 1	152	72

添加的注意力模块	AP	AP₅₀	AP₇₅	AP_S	AP_M	AP_L
基础网络^［26］	37.7	69.1	36.8	5.7	28.3	40.5
SE^［5］	39.3	70.0	39.3	7.4	28.4	44.7
CBAM^［12］	40.2	70.6	42.4	8.1	29.3	45.9
ECA^［14］	38.8	69.4	39.0	6.5	27.4	44.2
SimAM^［21］	38.9	69.3	39.5	9.0	28.5	44.0
文中模块	40.5	70.6	42.1	9.6	29.5	45.1

添加的模块	AP	AP₅₀	AP₇₅	AP_S	AP_M	AP_L
基础网络	37.7	69.1	36.8	5.7	28.3	40.5
FSM	38.9	69.4	38.3	6.9	28.5	44.6
FNM	39.5	70.1	40.4	9.1	28.8	44.4