YOLOv8-Based Real-Time Indonesian Sign Language (BISINDO) Detection System for Android Applications

Immanuel Batistuta Tarigan; Zulfahmi Indra; Mansur AS; Kana Saputra; Insan Taufik

doi:10.59934/jaiea.v6i1.2686

Authors

Immanuel Batistuta Tarigan Universitas Negeri Medan
Zulfahmi Indra Universitas Negeri Medan
Mansur AS Universitas Negeri Medan
Kana Saputra Universitas Negeri Medan
Insan Taufik Universitas Negeri Medan

DOI:

https://doi.org/10.59934/jaiea.v6i1.2686

Keywords:

Android Application; BISINDO; Conv1D; MediaPipe Hands; YOLOv8

Abstract

The manuscript should contain an abstract. The abstract should be self-contained and citation-free and should not exceed 200 words. The abstract should state the purpose, approach, results and conclusions of the work. The author should assume that the reader has some knowledge of the subject but has not read the paper. Thus, the abstract should be intelligible and complete in it-self (no numerical references); it should not cite figures, tables, or sections of the paper. The abstract should be written using third person instead of first person. Communication barriers between Deaf individuals and people unfamiliar with Indonesian Sign Language (BISINDO) reduce the effectiveness of daily interactions. This study develops a real-time BISINDO recognition system for Android by integrating YOLOv8, MediaPipe Hands, and a one-dimensional convolutional neural network (Conv1D). YOLOv8 is used to detect the user as the region of interest, while MediaPipe Hands extracts 21 hand landmarks that are processed by the Conv1D model to classify 15 BISINDO gesture classes. A balanced dataset consisting of 900 gesture samples was used for training and evaluation. The proposed architecture contains 59,680 trainable parameters, making it lightweight and suitable for mobile deployment. Experimental results show stable convergence during training, increasing training and validation accuracy, and decreasing loss values. The classification process correctly recognizes most gesture classes, with only limited confusion among visually similar gestures. The developed Android application performs real-time gesture recognition using a smartphone camera and converts recognized gestures into text. The proposed approach demonstrates that the integration of YOLOv8, MediaPipe Hands, and Conv1D provides an efficient and practical solution for real-time BISINDO recognition and has the potential to improve communication accessibility through mobile assistive technology.

Downloads

Download data is not yet available.

References

A. Kaisul Imam and K. Aeni, “Deteksi Sistem Isyarat Bahasa Indonesia (SIBI) Menggunakan Algoritma YOLOv5s,” Jurnal ..., 2023.

B. K. Pratama, S. Lestanti, and Y. Primasari, “Implementasi Algoritma You Only Look Once (YOLO) untuk Mendeteksi Bahasa Isyarat SIBI,” ProTekInfo (Pengembangan Riset dan Observasi Teknik Informatika), vol. 11, no. 2, pp. 7–14, 2024, doi: 10.30656/protekinfo.v11i2.9105.

H. M. Putri, F. Fadlisyah, and W. Fuadi, “Pendeteksian Bahasa Isyarat Indonesia Secara Real-Time Menggunakan Long Short-Term Memory (LSTM),” Jurnal Teknologi Terapan Sains 4.0, vol. 3, no. 1, 2022, doi: 10.29103/tts.v3i1.6853.

D. A. Primadiansyah et al., “Literature Review: Implementasi Machine Learning dalam Pengenalan Bahasa Isyarat Indonesia (BISINDO),” Jurnal ..., vol. 2, no. 5, pp. 821–824, 2024.

Y. Rahmawati, Y. Findawati, N. L. Azizah, and F. N. Hidayah, “Pengenalan Bahasa Isyarat Indonesia,” Infotek: Jurnal Informatika dan Teknologi, vol. 8, no. 2, pp. 323–333, 2025.

R. R. Agustin, H. Maulana, E. P. Mandyartha, and F. C. Science, “Detection of BISINDO (Indonesian Sign Language) Gestures into Text-to-Speech Using Long Short-Term Memory with MediaPipe Holistic,” vol. 5, no. 4, pp. 1051–1061, 2024.

S. Kiranyaz, O. Avci, O. Abdeljaber, T. Ince, M. Gabbouj, and D. J. Inman, “1D Convolutional Neural Networks and Applications: A Survey,” Mechanical Systems and Signal Processing, vol. 151, p. 107398, 2021, doi: 10.1016/j.ymssp.2020.107398.

T. Mahmud, M. C. A. Ngu, and G. Yang, “Why Android App Testing Falls Short: Empirical Insights from Open-Source Projects and a Practitioner Survey,” 2025.