Hybrid AI Model Detects Audio Deepfakes with 94% Accuracy

Published in Discover Informatics, the study introduces a system that combines Constant-Q Transform (CQT) scalograms with residual convolutional blocks, spatial self-attention, and transformer encoders. The model was evaluated on the benchmark Fake-or-Real (FoR) dataset, a large-scale corpus designed specifically for synthetic speech detection. By integrating time-frequency representations with advanced attention mechanisms, the framework aims to reveal subtle synthesis artifacts that are often imperceptible to the human ear. The results suggest that this multi-scale approach provides a stable decision framework for identifying speech spoofing attacks under standard evaluation protocols.

The motivation for this research stems from the rapid advancement of artificial intelligence technologies that facilitate the generation of realistic synthetic media. While early deepfake efforts focused primarily on visual manipulation, synthetic audio generation has emerged as a significant cybersecurity concern. Advanced neural text-to-speech and voice conversion systems can now create speech samples that are difficult to distinguish from genuine human voices. These capabilities pose risks to public figures, politicians, and everyday citizens, particularly as financial services and authentication systems increasingly rely on voice-based biometrics. The spread of sophisticated audio spoofing attacks presents a security challenge that directly impacts public safety and individual privacy, necessitating robust automated detection systems.

Traditional machine learning approaches, such as Gaussian Mixture Models and Support Vector Machines, have historically served as baselines in audio forensics but often struggle with high-dimensional acoustic complexities. Recent research has shifted toward deep learning architectures, including Convolutional Neural Networks and Recurrent Neural Networks, which capture localized spectro-temporal artifacts. More recently, advanced structures like Vision Transformers and Graph Neural Networks have been introduced to model complex contextual dependencies and long-range spectral relationships. Despite these advancements, existing detectors often remain vulnerable to feature representation bottlenecks and domain shifts. The study addresses this gap by investigating the representation power of various time-frequency acoustic features and proposing an optimized, attention-enhanced convolutional architecture.

The proposed framework begins with an acoustic frontend transformation process that converts raw audio waveforms into six distinct visual representations. These include the Short-time Fourier Transform spectrogram, Log Power Magnitude Spectrogram, Mel-Spectrogram, CQT Scalogram, Mel-frequency Cepstral Coefficients, and Constant-Q Cepstral Coefficients. The CQT Scalogram was selected as the primary frontend because it employs geometrically spaced frequency bins, providing finer frequency resolution at lower frequencies and higher temporal resolution at higher frequencies. This property makes the CQT well-suited for capturing harmonic structures and subtle synthetic speech artifacts. In a baseline comparison, the CQT Scalogram demonstrated the most reliable generalization, achieving an overall accuracy of 0.89 and exhibiting a balanced decision boundary between real and fake classes.

The architecture of the proposed network is structured into four main stages: local feature extraction, channel-spatial attention refinement, pixel-wise spatial self-attention, and global context modeling. The initial feature extraction phase processes the 3-channel visual feature through a hierarchical convolutional backbone consisting of three sequential stages. Each stage pairs a Residual Block with a Convolutional Block Attention Module (CBAM) followed by a Max-Pooling operation. The CBAM dynamically refines the feature maps by computing inter-channel relationships and focusing on where informative artifacts reside.

Following the convolutional backbone, the feature maps are passed into a Spatial Self-Attention module. While convolutional neural networks excel at capturing local patterns, synthetic audio artifacts often manifest as distributed anomalies across distant time-frequency bins. The spatial self-attention module projects the feature tensor into Queries, Keys, and Values, allowing every spatial region in the visual feature to evaluate its contextual affinity with all other regions. This step captures fine-grained structural correlations that local filters might miss. Subsequently, the token sequence is fed into two stacked Transformer Encoder layers to model long-range spectro-temporal dependencies over the entire representation space. These layers utilize multi-head self-attention mechanisms to capture complex, non-local harmonic interactions and global contextual cues.

The evaluation was conducted using the for-2sec subset of the Fake-or-Real dataset, which contains a total sample size of 17,870 utterances. The data was divided into 13,956 samples for training, 2,826 for validation, and 1,088 for testing, maintaining a perfect class balance between real and fake instances. To ensure rigorous generalization testing, the evaluation followed the official dataset partitioning defined by the original authors. The testing set was explicitly enforced at the speaker and algorithm level, containing exclusively unseen real voices and synthetic speech generated by an unseen text-to-speech algorithm. This isolation from the training and validation phases guarantees a realistic forensic evaluation scenario, preventing identity leakage and ensuring that the model’s performance reflects its ability to generalize to new data.

The proposed model achieved an overall test accuracy of 94%, an Area Under the Receiver Operating Characteristic Curve (AUROC) score of 0.9919, and an Equal Error Rate (EER) of 3.86%. A non-parametric bootstrapping analysis on the test set predictions yielded a stable mean accuracy of 93.88% with a 95% confidence interval spanning between 92.55% and 95.22%. The class-specific metrics revealed a balanced decision boundary, with a precision of 0.98 and a recall of 0.90 for the fake class. The high recall on real samples ensures that real speakers are rarely penalized by false alarms. The low EER indicates that the hybrid integration of residual convolutional feature extraction, spatial self-attention, and transformer encoders successfully minimizes decision errors against text-to-speech-generated voice spoofing attacks.

An ablation study was conducted to understand the individual contribution of each architectural component. The base residual network and the addition of CBAM yielded an identical accuracy of 0.89, indicating that CBAM alone does not alter overall classification capacity. Interestingly, integrating spatial self-attention without the transformer encoder resulted in a slight decrease in accuracy to 0.87, possibly due to functional overlap with the attention mechanisms already present in CBAM. However, when the transformer encoder was integrated alongside spatial self-attention, the accuracy increased to 0.94. This outcome suggests that spatial self-attention alone is insufficient to improve performance without the sequential modeling provided by the transformer layers, which help capture long-range dependencies across the scalogram representation.

The study also evaluated the model’s robustness under various audio degradation attacks. When subjected to additive white Gaussian noise at a 30 dB Signal-to-Noise Ratio, the model maintained a high classification accuracy of 0.86 and an EER of 7.90%. Against a median filtering attack, which introduces aggressive high-frequency digital smoothing, the architecture achieved an accuracy of 0.90 and an EER of 6.80%. However, performance degraded significantly under a physical re-recording attack, where utterances were played through a computer speaker and recorded with a non-professional microphone. In this scenario, accuracy dropped to 0.62, and the EER increased to 21.82%. This degradation highlights that re-recording introduces a complex domain shift that distorts the fine-grained synthetic artifacts the CQT scalograms rely on.

Explainability analysis using spatial attention maps revealed that the model concentrates attention in the lower-frequency harmonic region for fake samples. This region is particularly susceptible to distortion in neural text-to-speech pipelines, where vocoders often introduce subtle irregularities in the phase and amplitude envelope of low-frequency harmonics. In contrast, for real samples, the attention pattern shifts toward upper-frequency and vertical structures, corresponding to transient onsets and high-frequency energy bursts characteristic of natural speech production. These observations suggest that the model has learned to differentiate between spectral structures that are acoustically plausible in natural speech and those that carry synthesis artifacts, even when both occupy similar frequency bins.

The proposed architecture offers a favorable trade-off between computational efficiency and classification accuracy. The fully proposed model operates with only 624,000 parameters, a reduction of approximately 70% compared to the baseline convolutional network. While the inclusion of transformer encoder layers increases mathematical operations to 1.46 GFLOPs, the empirical inference latency is recorded at 4.03 ms per audio clip. This latency is virtually imperceptible in real-time human-computer interaction or live audio stream analysis. The study concludes that pairing time-frequency representations like CQT with sequential attention mechanisms provides a reliable countermeasure for forensic applications, although further validation across diverse recording environments and languages is needed to confirm generalizability.

Subject of Research: Audio deepfake detection using constant-Q transform scalograms and spatial-transformer attention networks

Article Title: Audio deepfake detection using constant-Q transform scalograms and spatial-transformer attention networks

Article References: Küçük, M., Şeker, Z. S., Çelik, B. N., & Sadık, Ş. A. (2026). Audio deepfake detection using constant-Q transform scalograms and spatial-transformer attention networks. Discover Informatics, 1(1), Article 38. https://doi.org/10.1007/s44564-026-00042-x

Image Credits: AI Generated

DOI: 10.1007/s44564-026-00042-x

Keywords: Audio, deepfake, detection, constant-Q, transform, scalograms, spatial-transformer, attention, networks