Improvement of Cross-Dataset Performance of Face Forgery Detection Based on Multi-Scale Spatiotemporal Features and Tampering Probabilities

doi:10.12141/j.issn.1000-565X.230105

Abstract

Abstract:

Most existing Deepfake face forgery detection algorithms suffer from the insufficient generalization performance despite that their intra-dataset detection performance is fairly good. This is because these methods mainly rely on local features that are prone to overfitting, which leads to unsatisfactory cross-dataset detection performance. In order to solve this problem, a face forgery detection method based on multi-scale spatiotemporal features and tampering probability is proposed, which helps to maintain good performance for cross-dataset testing, cross-forgery testing as well as video compression by detecting the inevitable temporal inconsistency between continuous frames in deepfake videos. The proposed detection method consists of three modules: a multi-scale spatiotemporal feature extraction module is employed to reveal the discontinuous traces of fake videos in the temporal domain, a three-dimension dual-attention module is designed to adaptively compute the correlation between multi-scale spatiotemporal features, and an auxiliary supervision module is used to predict the tampering probabilities of randomly selected pixels to form a supervision mask. Then, the proposed algorithm is compared with the baseline algorithm and the latest relevant works on large-scale public standard databases such as FF++, DFD, DFDC and CDF. Experimental results have show that the proposed algorithm has the best overall performance for cross-dataset testing and video compression, and has the above-average performance for cross-forgery testing. Meanwhile, it maintains good average performance for all intra-dataset testing. All the experiments demonstrate the effectiveness of the proposed algorithm.

Key words: face forgery detection, cross-dataset performance, multi-scale spatiotemporal feature, attention mechanism, tampering probability, 3D point cloud reconstruction

CLC Number:

TP391

HU Yongjian, ZHUO Sichao, LIU Beibei, WANG Yufei, LI Jicheng. Improvement of Cross-Dataset Performance of Face Forgery Detection Based on Multi-Scale Spatiotemporal Features and Tampering Probabilities[J]. Journal of South China University of Technology(Natural Science Edition), 2024, 52(6): 110-119.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Detailed structure of SFEM"

网络层	输出尺寸	DenseNet-121	输出通道
初步提取	224×224	1×1卷积	3
	112×112	7×7卷积	64
	56×56	最大池化	64
Dense Block1	56×56	$1 × 1 卷积 3 × 3 卷积$ × 6	256
Transition Block1	56×56	1×1×128卷积	128
Transition Block1	28×28	平均池化	128
Dense Block2	28×28	$1 × 1 卷积 3 × 3 卷积$ × 12	512
Transition Block2	28×28	1×1×256卷积	256
Transition Block2	14×14	平均池化	256

Table 1

Fig.5

Fig.6

Fig.7

Table 2

Table 3

Table 4

Table 5

Fig.8

Table 6

References 18

1	LI Y，LYU S ．Exposing DeepFake videos by detecting face warping artifacts［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：46-52．
2	LI J， XIE H， LI J，et al ．Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Virtual：IEEE，2021：6458-6467．
3	DURALL R， KEUPER M， KEUPER J ．Watch your up-convolution：CNN-based generative deep neural networks are failing to reproduce spectral distributions［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition．Seattle：IEEE，2020：7890-7899．
4	LI L， BAO J， ZHANG T，et al ．Face X-ray for more general face forgery detection［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Seattle：IEEE，2020：5001-5010．
5	胡永健，高逸飞，刘琲贝，等．基于图像分割网络的深度假脸视频篡改检测［J］．电子与信息学报，2021，43（1）：162-170．
	HU Yongjian， GAO Yifei， LIU Beibei，et al ．Deepfake videos detection based on image segmentation with deep neural networks［J］．Journal of Electronics & Information Technology，2021，43（1）：162-170．
6	YANG X， LI Y，LYU S ．Exposing deep fakes using inconsistent head poses［C］∥Proceedings of the 2019 IEEE International Conference on Acoustics，Speech and Signal Processing．Bredo：IEEE，2019：8261-8265．
7	HALIASSOS A， VOUGIOUKAS K， PETRIDIS S，et al ．Lips don’t lie：a generalisable and robust approach to face forgery detection［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Virtual：IEEE，2021：5039-5049．
8	AMERINI I， CALDELLI R ．Exploiting prediction error inconsistencies through LSTM-based classifiers to detect deepfake videos［C］∥Proceedings of the 2020 ACM Workshop on Information Hiding and Multimedia Security．Denver：ACM，2020：97-102．
9	MASI I， KILLEKAR A， MASCARENHAS R，et al ．Two-branch recurrent network for isolating deepfakes in videos［C］∥Proceedings of the 16th European Confe-rence on Computer Vision．Glasgow：Springer，2020：23-28．
10	ZHENG Y， BAO J， CHEN D，et al ．Exploring temporal coherence for more general video face forgery detection［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition．Virtual：IEEE，2021：15044-15054．
11	LI X， WAN J， JIN Y，et al ．3DPC-Net：3D point cloud network for face anti-spoofing［C］∥Proceedings of the 2020 IEEE International Joint Conference on Biometrics．Houston：IEEE，2020：1-8．
12	SHI X， CHEN Z， WANG H，et al ．Convolutional LSTM network：a machine learning approach for precipitation nowcasting［J］．Advances in Neural Information Processing Systems，2015，28：802-810．
13	ROSSLER A， COZZOLINO D， VERDOLIVA L，et al ．Faceforensics++：learning to detect manipulated facial images［C］∥Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition．Long Beach：IEEE，2019：1-11．
14	LIU H， LI X， ZHOU W，et al ．Spatial-phase shallow learning：rethinking face forgery detection in frequency domain［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Virtual：IEEE，2021：772-781．
15	LUO Y， ZHANG Y， YAN J，et al ．Generalizing face forgery detection with high-frequency features［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．Virtual：IEEE，2021：16317-16326．
16	CHEN S， YAO T， CHEN Y，et al ．Local relation learning for face forgery detection［C］∥Proceedings of the AAAI Conference on Artificial Intelligence．Virtual：AAAI，2021，35（2）：1081-1088．
17	NI Y， MENG D， YU C，et al ．CORE：consistent representation learning for face forgery detection［C］∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition．New Orleans：IEEE，2022：12-21．
18	SUN K， YAO T， CHEN S，et al ．Dual contrastive learning for general face forgery detection［C］∥Proceedings of the AAAI Conference on Artificial Intelligence．Virtual：AAAI，2022：2316-2324．

检测算法	不同数据库上的AUC/%			平均AUC/%
检测算法	DFD	DFDC	CDF	平均AUC/%
Xception^［13］	83.16	67.90	59.46	70.17
Face X-ray^［4］	85.60	70.01	74.20	76.94
SPSL^［14］	83.23	75.56	76.88	78.56
Two-Stream HF^［15］	91.90	79.70	79.40	83.73
LRL-Net^［16］	89.24	76.53	78.26	81.34
CORE^［17］	93.74	75.74	79.45	82.98
3D R50-FTCN^［10］	90.52	79.97	79.85	83.45
DCL^［18］	91.66	76.71	82.30	83.56
文中算法	95.37	85.31	81.43	87.37

伪造方法	检测算法	不同伪造方法下的AUC/%				平均AUC/%
伪造方法	检测算法	DF	F2F	FS	NT	平均AUC/%
DF	Xception^［13］	99.32	73.60	49.05	73.61	73.90
	Face X-ray^［4］	98.71	63.31	60.06	69.82	72.98
	Two-Stream HF^［15］	99.21	76.43	49.75	81.42	76.70
	DCL^［18］	99.98	77.13	61.01	75.01	78.28
	文中算法	99.95	77.66	53.25	84.14	78.75
F2F	Xception^［13］	80.33	99.47	76.25	69.66	81.50
	Face X-ray^［4］	45.82	98.15	96.12	94.57	87.47
	Two-Stream HF^［15］	83.71	99.45	98.77	98.46	95.10
	DCL^［18］	91.91	99.21	59.58	66.67	79.34
	文中算法	88.25	99.89	85.29	81.24	88.67
FS	Xception^［13］	66.45	88.83	99.40	71.32	81.43
	Face X-ray^［4］	63.02	98.44	93.83	94.57	83.96
	Two-Stream HF^［15］	68.80	99.38	99.54	98.01	91.43
	DCL^［18］	74.80	69.75	99.90	52.60	74.26
	文中算法	57.28	89.46	99.82	71.61	79.54
NT	Xception^［13］	79.98	81.36	73.17	99.15	83.42
	Face X-ray^［4］	70.51	91.77	91.03	92.54	86.46
	Two-Stream HF^［15］	89.40	99.52	93.35	99.46	96.93
	DCL^［18］	91.23	52.13	79.31	98.97	80.41
	文中算法	93.81	94.08	94.77	99.91	93.14

检测算法	ACC/%		AUC/%
检测算法	FF++（c23）	FF++（c40）	FF++（c23）	FF++（c40）
Xception^［13］	95.73	86.86	96.30	89.30
Face X-ray^［4］	—	—	87.40	61.60
SPSL^［14］	91.50	81.57	95.32	82.82
Two-Stream HF^［15］	97.74	88.95	99.36	94.10
LRL-Net^［16］	97.59	91.47	99.46	95.21
CORE^［17］	97.61	87.99	99.66	90.61
3D R50-FTCN^［10］	96.65	90.72	99.23	94.78
DCL^［18］	93.58	89.95	99.30	94.94
文中算法	97.76	91.48	99.70	95.56

编号	训练集AUC/%			测试集AUC/%
编号	MSTE	3D-DAM	PTPVR	DFD	DFDC	CDF
A1			√	93.78	84.01	74.77
A2	√		√	93.62	82.14	76.61
A3	√	√		93.30	80.01	78.71
A4	√	√	√	95.37	85.31	81.43

编号	方法	测试集AUC/%
编号	方法	DFD	DFDC	CDF
B1	MSTE+3D-DAM	93.30	80.01	78.71
B2	MSTE+3D-DAM+FCN	91.14	78.56	76.26
B3	MSTE+3D-DAM+PTPVR	95.37	85.31	81.43