Small-Sample Fault Diagnosis Method Based on Multi-Head Convolution and Differential Self-Attention

doi:10.12141/j.issn.1000-565X.220626

Abstract

Abstract:

Bearing is one of the most widely used rotating parts in industrial equipment. If the bearing runs in fault condition for a long time, it will cause huge economic loss and threaten personal safety, so that the investigation of bearing fault diagnosis is of great significance. Fault diagnosis technology based on deep learning is becoming more and more mature, but there are problems such as over-fitting, unstable effect and low accuracy in the case of small samples. In order to solve these problems, this paper proposes a Transformer variant model MDT (Multi-Head Convolution and Differential Self-Attention Transformer) to realize end-to-end few-shot fault diagnosis. This model combines the new data embedding algorithm of MC (Multi-Head Convolution) and the DSA (Differential Self-Attention) mechanism. The MC algorithm performs multi-path one-dimension convolution on the sample, extends the sample from one dimension to two dimensions by multi-channel output, and extracts rich fault information in each frequency domain in the original sample through multiple convolution kernel sizes. As compared with the original dot product self-attention in Transformer, the DSA mechanism obtains the corresponding attention weight vector for each feature through the difference, so as to extract deeper fault features from the sample. MDT inherits the powerful ability of Transformer to process sequence data, which can extract richer fault information from time-domain signals and avoid the overfitting problem common in small-sample models. Experimental results show that the proposed method can stably obtain more than 99% test accuracy in the bearing fault diagnosis task with only 100 training samples per fault type, and has strong anti-overfitting ability and strong robustness.

Key words: multi-head convolution, differential self-attention, Transformer variant, small sample, fault diagnosis

CLC Number:

TH133.3

CHEN Xindu, FU Zhisen, WU Zhiheng, et al.. Small-Sample Fault Diagnosis Method Based on Multi-Head Convolution and Differential Self-Attention[J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(7): 21-33.

Figures/Tables 23

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Table 1

Fig.9

Table 2

Table 3

Table 4

Table 5

Table 6

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Table 7

Table 8

References 17

1	赵玉成，陈荣华，马占国．旋转机械动力辨识与故障诊断技术［M］．徐州：中国矿业大学出版社，2008．
2	SHAO H， JIANG H， ZHANG X，et al ．Rolling bearing fault diagnosis using an optimization deep belief network［J］．Measurement Science and Technology，2015，26（11）：115002．
3	GOODFELLOW I， POUGET-ABADIE J， MIRZA M，et al ．Generative adversarial networks［J］．Communications of the ACM，2020，63（11）：139-144．
4	RADFORD A， METZ L， CHINTALA S ．Unsupervised representation learning with deep convolutional generative adversarial networks［DB/OL］．（2016-01-17）［2022-08-04］．．
5	GULRAJANI I， AHMED F， ARJOVSKY M，et al ．Improved training of Wasserstein GANs［C］∥NIPS’17：Proceedings of the 31st International Conference on Neural Information Processing Systems，2017．［S. l.］：［s. n.］，2017：5767-5777．
6	SHEN S， JIN G， GAO K，et al ．AE-GAN：adversarial eliminating with GAN［DB/OL］．（2017-09-26）［2022-08-04］．．
7	MIRZA M， OSINDERO S ．Conditional generative adversarial nets［DB/OL］．（2014-11-06）［2022-08-04］．．
8	SAUFI S R， AHMAD Z A B， LEONG M S，et al ．Gearbox fault diagnosis using a deep learning model with limited data sample［J］．IEEE Transactions on Industrial Informatics，2020，16（10）：6263-6271．
9	LI X， ZHANG W， DING Q ．Understanding and improving deep learning-based rolling bearing fault diagnosis with attention mechanism［J］．Signal Processing，2019，161：136-154．
10	ZHANG X， HE C， LU Y，et al ．Fault diagnosis for small samples based on attention mechanism［J］．Measurement，2022，187：110242．
11	XIE Z， CHEN J， FENG Y，et al ．End to end multi-task learning with attention for multi-objective fault diagnosis under small sample［J］．Journal of Manufacturing Systems，2022，62：301-316．
12	VASWANI A， SHAZEER N， PARMAR N，et al ．Attention is all you need［C］∥ Proceedings of the 31st Conference on Neural Information Processing Systems （NIPS 2017）．Long Beach：［s. n.］，2017．
13	SHAZEER N， MIRHOSEINI A， MAZIARZ K，et al ．Outrageously large neural networks：the sparsely-gated mixture-of-experts layer［DB/OL］．（2017-03-04）［2022-08-04］．．
14	DING Y， JIA M， MIAO Q，et al ．A novel time-frequency Transformer based on self-attention mechanism and its application in fault diagnosis of rolling bearings［J］．Mechanical Systems and Signal Processing，2022，168：108616．
15	郑英，金淼，张洪，等．一种基于一维多路卷积神经网络的故障分类方法：CN110033021A［P］．2019-07-19．
16	DAUBECHIES I， LU J， WU H T ．Synchrosqueezed wavelet transforms：an empirical mode decomposition-like tool［J］．Applied and Computational Harmonic Analysis，2011，30（2）：243-261．
17	HUANG N E， SHEN Z， LONG S R，et al ．The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis［J］．Proceedings of the Royal Society of London. Series A：Mathematical，Physical and Engineering Sciences，1998，454（1971）：903-995．

编号	头数	卷积核大小	卷积层数	步长	输出通道	训练准确率/%	测试准确率/%
1	2	21，15	3	1，1，1	80，70，48	96.00	94.45
2	2	21，11	4	2，2，2，2	80，70，48，48	96.80	96.02
3	2	21，7	4	4，2，2，2	80，70，48，48	97.10	95.73
4	3	21，15，11	3	1，1，1	80，70，32	99.14	98.85
5	3	21，15，7	4	2，2，2，2	80，70，32，32	99.83	99.60
6	3	21，15，11	4	4，2，2，2	80，70，32，32	99.70	99.78
7	4	21，15，11，7	3	1，1，1	80，70，24	98.90	98.54
8	4	21，15，11，7	4	2，2，2，2	80，70，24，24	99.37	99.21
9	4	21，15，11，7	4	4，2，2，2	80，70，24，24	99.28	99.46

嵌入法	训练准确率/%	测试准确率/%
SSWT	69.95	65.32
EMD	63.88	67.67
多头卷积嵌入法	99.70	99.78

实验编号	头数	块数	批量大小	样本长度	嵌入维度	训练准确率/%	测试准确率/%
1	3	2	150	1 024	64	95.97	94.51
2	3	4	100	2 480	96	98.86	96.72
3	3	3	150	4 960	96	97.98	96.90
4	4	2	150	1 024	64	99.20	95.84
5	4	4	100	2 480	96	99.43	98.40
6	4	3	150	4 960	96	100.00	99.78
7	5	2	150	1 024	64	97.50	89.37
8	5	4	100	2 480	96	97.99	90.61
9	5	3	150	4 960	96	98.91	92.46

超参数	取值		取值
批量大小	150	编码器堆叠数目	3
初始学习率	0.001	Dropout	0.1
权重衰减	0.000 1	卷积头数	3
样本长度	4 960	卷积核尺寸	21，15，11
训练数据尺寸	［156，96］	分类器神经元数	60，10
注意力头数	4

模型	平均准确率/%	最大准确率/%	方差	耗时/s
MDT	99.79	99.92	1.55×10^-6	881.4
DNN	84.17	96.21	9.13×10^-3	430.8
ResNet-CNN	89.49	97.06	1.32×10^-4	246.7
GRU	90.86	93.94	5.25×10^-3	369.3