AdfNet: An Adaptive Deep Forgery Detection Network Based on Diverse Features

doi:10.12141/j.issn.1000-565X.220825

Abstract

Abstract:

The harm caused by video tampering has been endangering people’s lives, which makes deep forgery detection technology gradually obtain widespread attention and development. However, current detection methods could not effectively capture noisy residuals due to the use of inflexible constraints. In addition, they ignore the correlation between texture and semantic features and the impact of temporal features on detection performance improvement. To solve these problems, this paper proposed an adaptive network (AdfNet) with diverse features for deep forgery detection. It helps the classifier to judge authenticity by extracting semantic features, texture features and temporal features. The paper explored the adaptive texture noise extraction (ATNEM) mechanism, and flexibly captured the noise residuals in non-fixed frequency bands through unpooled feature mapping and frequency-based channel attention mechanism. The deep semantic analysis guidance strategy (DSAGS) was designed to highlight the tampering traces through spatial attention mechanism, and guide the feature extractor to focus on the deep features of the focus region. The paper studied multi-scale temporal feature processing (MTFPM), and used temporal attention mechanism to assign weights to different video frames and capture the difference of time series in tampered videos. The experimental results show that the ACC score of the proposed network in the HQ mode of FaceForensics++(FF++) dataset is 97.41%, which is significantly better than that of the existing mainstream algorithms. Moreover, while maintaining the AUC value of 99.80% on the FF++ dataset, the AUC value can reach 76.41% on Celeb-DF, reflecting strong generalization.

Key words: deep learning, deep forgery detection, multi-scale temporal feature, attention mechanism, adaptive network

CLC Number:

TP391.4

LI Jiachun, LI Bowen, LIN Weiwei. AdfNet: An Adaptive Deep Forgery Detection Network Based on Diverse Features[J]. Journal of South China University of Technology(Natural Science Edition), 2023, 51(9): 82-89.

Figures/Tables 9

Fig.1

Fig. 2

Fig.3

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

References 27

1	KORSHUNOV P， MARCEL S ．Deepfakes：a new threat to face recognition？assessment and detection［EB/OL］．（2018-12-20）［2022-09-01］．.
2	胡永健， SALMAN Alhamidi，王宇飞，等．视频篡改检测数据库的构建及测试［J］．华南理工大学学报（自然科学版），2017，45（12）：57-64.
	HU Yong-jian， SALMAN Alhamidi， WANG Yu-fei，et al ．Construction and evaluation of video forgery detection database［J］．Journal of South China University of Technology （Natural Science Edition），2017，45（12）：57-64.
3	WANG J， SUN Y， TANG J ．LISIAM：Localization invariance Siamese network for deepfake detection［J］．IEEE Transactions on Information Forensics and Security，2022，17：2425-2436.
4	陆璐，钟文煜，吴小坤．基于多尺度视觉Transformer的图像篡改定位［J］．华南理工大学学报（自然科学版），2022，50（6）：10-18.
	LU Lu， ZHONG Wenyu， WU Xiaokun ．Image tampering localization based on mutil-scale transformer［J］．Journal of South China University of Technology（Natural Science Edition），2022，50（6）：10-18.
5	YANG J， XIAO S， LI A，et al ．Detecting fake images by identifying potential texture difference［J］．Future Generation Computer Systems，2021，125：127-135.
6	COZZOLINO D， POGGI G， VERDOLIVA L ．Recasting residual-based local descriptors as convolutional neural networks：An application to image forgery detection［C］∥Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security．Philadelphia：ACM，2017：159-164.
7	BAYAR B， STAMM M C ．A deep learning approach to universal image manipulation detection using a new convolutional layer［C］∥Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security．Vigo：ACM，2016：5-10.
8	AFCHAR D， NOZICK V， YAMAGISHI J，et al ．Mesonet：A compact facial video forgery detection network［C］∥Proceedings of the 2018 IEEE International Workshop on Information Forensics and Security （WIFS）．Hong Kong：IEEE，2018：1-7.
9	CHOLLET F ．Xception：Deep learning with depthwise separable convolutions［C］∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu：IEEE，2017：1251-1258.
10	MASI I， KILLEKAR A， MASCARENHAS R M，et al ．Two-branch recurrent network for isolating deepfakes in videos［C］∥Proceedings of the European Conference on Computer Vision．Glasgow：Springer，2020：667-684.
11	ZHAO H， ZHOU W， CHEN D，et al ．Multi-attentional deepfake detection［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Kuala Lumpur：IEEE，2021：2185-2194.
12	WU X， XIE Z， GAO Y T，et al ．Sstnet：Detecting manipulated faces through spatial，steganalysis and temporal features［C］∥Proceedings of the 2020 IEEE International Conference on Acoustics，Speech and Signal Processing （ICASSP）．Barcelona：IEEE，2020：2952-2956.
13	GÜERA D， DELP E J ．Deepfake video detection using recurrent neural networks［C］∥Proceedings of the 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance （AVSS）．Auckland：IEEE，2018：1-6.
14	SABIR E， CHENG J， JAISWAL A，et al ．Recurrent convolutional strategies for face manipulation detection in videos［J］．Interfaces （GUI），2019，3（1）：80-87.
15	ROSSLER A， COZZOLINO D， VERDOLIVA L，et al ．FaceForensics++：Learning to detect manipulated facial images［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Calif：IEEE，2019：1-11.
16	LI Y， YANG X， SUN P，et al ．Celeb-DF：A large-scale challenging dataset for deepfake forensics［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：3207-3216.
17	DENG J， DONG W， SOCHER R，et al ．ImageNet：A large-scale hierarchical image database［C］∥Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition．Miami：IEEE，2009：248-255.
18	BOROUMAND M， CHEN M， FRIDRICH J ．Deep residual network for steganalysis of digital images［J］．IEEE Transactions on Information Forensics and Security，2018，14（5）：1181-1193.
19	QIN Z， ZHANG P， WU F，et al ．Fcanet：Frequency channel attention networks［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision．Montreal：IEEE，2021：783-792.
20	LIU H， LI X， ZHOU W，et al ．Spatial-phase shallow learning：Rethinking face forgery detection in frequency domain［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Kuala Lumpur：IEEE，2021：772-781.
21	QIAN Y， YIN G， SHENG L，et al ．Thinking in frequency：Face forgery detection by mining frequency-aware clues［C］∥Proceedings of the European Conference on Computer Vision．Glasgow：Springer，2020：86-103.
22	TAN M， LE Q ．Efficientnet：Rethinking model scaling for convolutional neural networks［C］∥Proceedings of the International Conference on Machine Learning. New York：PMLR，2019：6105-6114.
23	TRINH L， TSANG M， RAMBHATLA S，et al ．Interpretable and trustworthy deepfake detection via dynamic prototypes［C］∥Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision．Montreal：IEEE，2021：1973-1983.
24	NGUYEN H H， FANG F， YAMAGISHI J，et al ．Multi-task learning for detecting and segmenting manipulated facial images and videos［C］∥Proceedings of the 2019 IEEE 10th International Conference on Biometrics Theory，Applications and Systems （BTAS）．Tampa：IEEE，2019：1-8.
25	CHO K， VAN M B， GULCEHRE C，et al ．Learning phrase representations using RNN encoder-decoder for statistical machine translation［EB/OL］．（2014-09-03）［2022-09-01］．.
26	LIN T Y， ROYCHOWDHURY A， MAJI S ．Bilinear CNN models for fine-grained visual recognition［C］∥Proceedings of the IEEE International Conference on Computer Vision．Santiago：IEEE，2015：1449-1457.
27	DAI Y， GIESEKE F， OEHMCKE S，et al ．Attentional feature fusion［C］∥Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision．Montreal：IEEE，2021：3560-3569.

方法	LQ		HQ
方法	ACC	AUC	ACC	AUC
MesoNet^［8］	70.47	—	83.10	—
SRNet^［17］	73.56	78.60	95.02	97.04
Xception^［9］	85.80	88.68	94.27	96.41
文献［13］	85.32	87.23	94.47	95.68
文献［14］	87.10	89.25	95.07	97.52
文献［10］	—	86.95	—	98.70
文献［11］	87.03	88.36	96.68	98.99
文献［12］	87.25	88.79	95.61	98.12
文献［20］	81.57	82.82	91.50	95.32
文献［3］	87.81	85.39	94.90	98.37
AdfNet	88.43	91.53	97.41	99.38

方法	FF++	Celeb-DF
MesoNet^［8］	84.70	54.80
Xception-c23^［9］	99.70	65.30
文献［10］	93.18	73.41
文献［21］	98.10	65.17
文献［22］	99.70	64.29
文献［11］	99.80	67.44
文献［20］	96.91	76.88
文献［23］	99.20	68.20
文献［24］	76.30	54.30
AdfNet	99.80	76.41

训练集	方法	测试集
训练集	方法	DF	F2F	FS	NT	FSH
DF	MesoNet	97.63	46.55	60.70	77.49	69.74
	Xception	99.31	74.56	38.69	75.36	62.88
	AdfNet	99.85	73.95	65.28	84.47	65.52
F2F	MesoNet	74.00	97.78	53.57	63.67	54.13
	Xception	81.74	99.38	57.94	69.08	62.59
	AdfNet	86.34	99.60	61.95	67.68	65.71
FS	MesoNet	51.19	58.18	98.95	51.27	49.23
	Xception	58.35	67.36	99.44	54.26	64.68
	AdfNet	60.41	66.85	99.71	58.38	57.39
NT	MesoNet	87.54	61.92	49.30	86.86	63.88
	Xception	84.65	69.78	46.83	98.19	70.55
	AdfNet	93.63	70.96	50.76	98.65	70.77
FSH	MesoNet	78.33	45.83	47.90	65.31	95.13
	Xception	74.13	47.36	50.39	64.62	99.35
	AdfNet	64.36	52.18	52.20	65.90	99.49

方法	DF（HQ）		F2F（HQ）		FS（HQ）		NT（HQ）		FSH（HQ）
方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
双流网络	98.13	98.67	97.87	98.01	98.01	98.83	93.80	97.99	97.07	99.43
双流网络+双向GRU	98.59	99.36	98.78	99.34	98.97	99.33	94.09	98.65	97.64	99.49
双流网络+MTFPM	98.92	99.80	98.80	99.53	98.95	99.74	94.48	98.52	97.76	99.63
双流网络+ATNEM	98.64	99.70	98.88	99.56	98.79	99.57	94.34	98.46	97.12	99.51
双流网络+DSAGS	98.77	99.76	98.75	99.50	98.55	99.54	94.19	98.15	97.41	99.55
AdfNet	99.15	99.87	98.99	99.60	99.12	99.75	94.85	98.67	97.90	99.49

伪造方法	单帧		连续两帧		连续3帧		连续4帧
伪造方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
DF	98.13	99.63	99.15	99.87	99.23	99.68	99.40	99.87
F2F	98.36	99.24	98.99	99.60	98.73	99.61	98.99	99.52
FS	98.51	99.72	99.12	99.75	99.13	99.82	99.05	99.77
NT	94.29	98.08	94.85	98.67	94.69	98.54	95.03	98.50
FSH	96.89	99.47	97.90	99.49	97.88	99.53	98.01	99.64