Voice Deepfake Detection Using Spectrogram Features and Mel-Frequency Cepstral Coefficients with a Combination of Convolutional Neural Network and Recurrent Neural Network

Ade Setiawan; Hermawan Syahputra; Yulita Molliq Rangkuti; Zulfahmi Indra; Kana Saputra S

doi:10.59934/jaiea.v5i3.2507

Authors

Ade Setiawan Universitas Negeri Medan
Hermawan Syahputra Universitas Negeri Medan
Yulita Molliq Rangkuti Universitas Negeri Medan
Zulfahmi Indra Universitas Negeri Medan
Kana Saputra S Universitas Negeri Medan

DOI:

https://doi.org/10.59934/jaiea.v5i3.2507

Keywords:

Voice deepfake, Audio detection, Convolutional Neural Network, Mel-Frequency Cepstral Coefficients, Recurrent Neural Network, Spectrogram

Abstract

The rapid development of artificial intelligence technology has driven the emergence of voice deepfakes that are increasingly realistic and difficult to distinguish from genuine human speech. This condition creates significant risks, including identity misuse, fraud, and information manipulation, thereby requiring an effective detection system capable of identifying manipulated voice patterns with high accuracy. This study aims to develop a voice deepfake detection system by combining Convolutional Neural Network (CNN) and Recurrent Neural Network (BiLSTM) methods. Audio features were extracted using Mel-Frequency Cepstral Coefficients (MFCC) and spectrograms to capture both frequency characteristics and temporal dynamics simultaneously. All genuine and deepfake voice recordings underwent preprocessing stages, including noise reduction, normalization, segmentation, and augmentation, to improve data diversity and model robustness. The model was evaluated using several data split ratios to determine the most optimal performance. The best result was achieved with an 80:20 ratio, reaching an accuracy of 99.3% and an AUC value of 0.9999. These results demonstrate the model’s strong capability to identify subtle structural changes in audio signals that are difficult to detect using conventional methods. Based on these findings, the CNN–RNN (BiLSTM) approach proved to be highly effective for detecting manipulated voice recordings. This research provides an important contribution to the development of audio security systems and the mitigation of risks associated with the misuse of deepfake technology across various sectors.

Downloads

Download data is not yet available.

References

S. Widya Sasana, “Etika Penggunaan Teknologi AI Menurut Paul Ricoeur sebagai Realisasi Hidup Baik,” Paradigma, vol. 30, no. 1, pp. 17–27, 2023. doi: 10.33503/paradigma.v30i1.4020.

R. Angelika Septi Rahayu and H. Santoso, “Analisis Gambar Wajah Palsu: Mendeteksi Keaslian Gambar yang Dimanipulasi Menggunakan Metode Variasional Autoencoder dan Forensik Jaringan Neural Deep,” Sibatik Journal, vol. 2, no. 9, 2023. doi: 10.54443/sibatik.v2i9.1312.

Z. Khanjani, G. Watson, and V. P. Janeja, “Audio Deepfakes: A Survey,” Frontiers in Big Data, vol. 5, p. 1001063, 2023. doi: 10.3389/fdata.2022.1001063.

C. Stupp, “Fraudsters Used AI to Mimic CEO’s Voice in Unusual Cybercrime Case,” WSJ Pro Cyber Security, Aug. 30, 2019.

Kementerian Komunikasi dan Informatika, “Antisipasi Deep Fake, Wamen Nezar Patria: Kominfo Lindungi Kelompok Rentan,” 2023. [Online]. Available: https://www.kominfo.go.id/

K. T. Mai, S. Bray, T. Davies, and L. D. Griffin, “Warning: Humans Cannot Reliably Detect Speech Deepfakes,” PLOS ONE, vol. 18, no. 8, p. e0285333, 2023. doi: 10.1371/journal.pone.0285333.

B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The DeepFake Detection Challenge (DFDC) Dataset,” 2020. [Online]. Available: http://arxiv.org/abs/2006.07397

B. Malolan, A. Parekh, and F. Kazi, “Explainable Deep-Fake Detection Using Visual Interpretability Methods,” in Proc. 3rd Int. Conf. Information and Computer Technologies (ICICT), 2020, pp. 289–293. doi: 10.1109/ICICT50521.2020.00051.

S. Y. Lim, D. K. Chae, and S. C. Lee, “Detecting Deepfake Voice Using Explainable Deep Learning Techniques,” Applied Sciences, vol. 12, no. 8, 2022. doi: 10.3390/app12083926.

M. U. Tanveer, K. Munir, M. Amjad, A. U. Rehman, and A. Bermak, “Unmasking the Fake: Machine Learning Approach for Deepfake Voice Detection,” IEEE Access, 2024. doi: 10.1109/ACCESS.2024.3521026.

L. Gaur, DeepFakes. Boca Raton: CRC Press, 2022. doi: 10.1201/9781003231493.

S. Karnouskos, “Artificial Intelligence in Digital Media: The Era of Deepfakes,” IEEE Transactions on Technology and Society, vol. 1, no. 3, pp. 138–147, 2020. doi: 10.1109/TTS.2020.3001312.

S. Y. Hartono, S. Suyuti, Bambang, P. Asmara, A. Ashad, and N. K. Hamzidah, “Analisis Spektogram Sinyal Suara Asli dan Suara Hasil Konversi Berbasis Derivative Gelombang Glotal,” Jurnal Teknologi Elekterika, vol. 20, no. 2, pp. 24–28, 2023.

P. Stoica and R. Moses, Spectral Analysis of Signals. Upper Saddle River, NJ: Pearson Prentice Hall, 2005.

S. Ali, S. Tanweer, S. Khalid, and N. Rao, “Mel Frequency Cepstral Coefficient: A Review,” Mar. 17, 2021. doi: 10.4108/eai.27-2-2020.2303173.

E. Grossi and M. Buscema, “Introduction to Artificial Neural Networks,” European Journal of Gastroenterology and Hepatology, vol. 19, no. 12, pp. 1046–1054, 2007. doi: 10.1097/MEG.0b013e3282f198a0.

A. Raup, W. Ridwan, Y. Khoeriyah, Q. Yuliati Zaqiah, and U. Islam Negeri Sunan Gunung Djati Bandung, “Deep Learning dan Penerapannya dalam Pembelajaran,” 2022.

S. Ilahiyah and A. Nilogiri, “Implementasi Deep Learning Pada Identifikasi Jenis Tumbuhan Berdasarkan Citra Daun Menggunakan Convolutional Neural Network,” 2018.

R. Onsu, D. Febrian, and F. Diane, “Implementasi Bi-LSTM Dengan Ekstraksi Fitur Word2vec Untuk Pengembangan Analisis Sentimen Aplikasi Identitas Kependudukan Digital,” Jurnal Teknologi Terpadu, vol. 10, no. 1, pp. 46–55, 2024.

J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venugopalan, S. Guadarrama, K. Saenko, and T. Darrell, “Long-term Recurrent Convolutional Networks for Visual Recognition and Description,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 4, pp. 677–691, 2014. doi: 10.1109/TPAMI.2016.2599174.

M. I. Abidin, I. Nurtanio, and A. Achmad, “Deepfake Detection in Videos Using Long Short-Term Memory and CNN ResNext,” ILKOM Jurnal Ilmiah, vol. 14, no. 3, pp. 285–292, 2022. doi: 10.33096/ilkom.v14i3.1254.

H. Hermanto and T. W. Sen, “Syllable-Based Javanese Speech Recognition Using MFCC and CNNs: Noise Impact Evaluation,” Jurnal Teknik Informatika, vol. 18, no. 1, 2025. doi: 10.15408/jti.v18i1.41067.

A. Adila, C. O. Mawalim, and M. Unoki, “Detecting Spoof Voices in Asian Non-Native Speech: An Indonesian and Thai Case Study,” arXiv preprint arXiv:2412.01040, 2024.