Audio–Visual Synchronization and Lip Movement Analysis for Real-Time Deepfake Detection
Muhammad Yaqoob Javed, Zhaohui Zhang, Fida Hussain Dahri, Asif Ali Laghari, Martin Krajčík, Ahmad Almadhor
Abstract
The rapid advancements in Artificial Intelligence (AI) and deep learning techniques have led to the creation of highly realistic synthetic media known as deepfakes. These manipulated images, videos, and audio pose significant ethical, security, and privacy concerns. To address this issue, we propose a novel Audio–Visual Synchronisation and Fusion Framework (AVSFF) for real-time detection of deepfakes. This approach focuses on fine-grained lip movement analysis by detecting subtle inconsistencies between lip movements and corresponding audio. By integrating visual and audio features using multimodal fusion techniques, AVSFF aims to distinguish between authentic and manipulated media. The proposed framework is evaluated on diverse datasets, including FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF, demonstrating promising results with accuracies of 0.9973, 0.9760, 0.9890, and 0.9786, respectively. This study contributes to the field by providing a robust real-time solution for detecting deepfakes in audio–visual synthetic data, ensuring enhanced detection accuracy and effective generalization across various deepfake manipulations and demographic data.