AI-DRIVEN SOFTWARE SYSTEM FOR TRANSLATING SPOKEN UKRAINIAN INTO SIGN LANGUAGE
DOI:
https://doi.org/10.17721/3041-2323.2025.137-164Keywords:
inclusive and assistive technologies, Ukrainian Sign Language (USL), artificial intelligence, automatic speech recognition (ASR), Whisper model, lemmatization, JSON-based error correction, GPU CUDA accelerationAbstract
This paper presents a software system designed for real-time translation of spoken Ukrainian into sign language animations. The system employs the Whisper model for automatic speech recognition (ASR), ensuring reliable transcription of oral Ukrainian. Given that the Ukrainian language is highly inflected while sign language does not utilize grammatical cases, a JSON-based correction module has been implemented. This module consists of two components: a dictionary for frequent recognition errors and a lemmatization mechanism that converts words into their base forms. The normalized text is subsequently mapped to GIF animations, each representing a corresponding sign in Ukrainian Sign Language (USL). The translation process operates in real time, providing a sequential display of animations synchronized with the recognized speech. The solution demonstrates both technical efficiency—characterized by low latency and a modular architecture—and social impact by enhancing communication accessibility for individuals who are d/Deaf or hard of hearing, as well as those who use sign language as their primary means of communication. Potential applications include inclusive education, everyday interactions, and the development of assistive technologies tailored to the needs of the signing community.
References
Alphacephei. (n.d.). Vosk speech recognition toolkit: Vosk API documentation. https://alphacephei.com/vosk/
Camgoz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020). Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10023–10033).
Dreuw, P., Neidle, C., Athitsos, V., Sclaroff, S., & Ney, H. (2007). Benchmark databases for video-based automatic sign language recognition. In Proceedings of the 6th International Conference on Language Resources and Evaluation (pp. 1558–1563).
Elliott, R., Glauert, J.R.W., Kennaway, J.R., Marshall, I., & Sáfár, É. (2008). Linguistic modeling and language-processing technologies for avatar-based sign language presentation. Universal Access in the Information Society, 6(4), 375–391. https://doi.org/10.1007/s10209-007-0102-z
Efthimiou, E., Fotinea, S. E., Dimou, A. L., & Karioris, P. (2009). Developing an eLearning platform for the Greek sign language. In Proceedings of the International Conference on eLearning and Accessibility for the Disabled (eLAD 2009) (pp. 43–50). Hellenic Open University.
Krak, Iu., Barmak, O., & Romanyshyn, S. (2017). Information technology for automated translation from inflected languages to sign language. In W. Wojcik & J. Sikora (Eds.), Recent advances in information technology (pp. 35–48). CRC Press.
López-Ludeña, V., San-Segundo, R., Montero, J. M., Barra-Chicote, R., & Lorenzo, J. (2014). Architecture of a Spanish into sign language translation system. In Proceedings of the 9th International Conference on Language Resources and Evaluation (pp. 2819–2823). European Language Resources Association.
Moryossef, A., Yin, K., Neubig, G., & Goldberg, Y. (2021). Data augmentation for sign language gloss translation. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (pp. 1–11).
Othman, A., & Jemni, M. (2012). Statistical sign language machine translation: From English written text to American sign language gloss. International Journal of Computer Science Issues, 9(4), 65–73.
Prystupa, M., & Sydorenko, T. (2019). Morphological complexity in the Ukrainian language and its implications for NLP. In Proceedings of the International Conference on Computational Linguistics (COLING). Association for Computational Linguistics. https://aclanthology.org/
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Whisper: Robust speech recognition via large-scale weak supervision [OpenAI Technical Report]. https://cdn.openai.com/papers/whisper.pdf
Saunders, B., Camgoz, N. C., & Bowden, R. (2020). Progressive transformers for end-to-end sign language production. In European Conference on Computer Vision (pp. 687–705). Springer.
Speers, D. (2001). Representation of American Sign Language for machine translation [Doctoral dissertation, Georgetown University].
Stein, D., Bungeroth, J., & Ney, H. (2006). Morpho-syntax based statistical methods for sign language translation. In Proceedings of the 11th Annual Conference of the European Association for Machine Translation (pp. 169–177).
Sutton, V. (2019). SignWriting: Read and write any sign language. SignWriting Press.
Ukrainian Society of the Deaf (UTOG). (2020). Ukrainian Sign Language dictionary. https://utog.org/
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/1706.03762
Yin, K., & Read, J. (2020). Better sign language translation with STMC-transformer. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 5975–5989).
Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N., & Palmer, M. (2000). A machine translation system from English to American Sign Language. In Conference of the Association for Machine Translation in the Americas (pp. 54–67). Springer.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Applied information systems and technologies in the digital society
This work is licensed under a Creative Commons Attribution 4.0 International License.
