Sign language is expressed through movements and position of hands and other body parts. Although hearing people often think that sign language is international, it has different grammar and lexicon in every language. Today’s market lacks systems that could recognise not only sign alphabet, but also other signs characteristic to the sign language.
The first Lithuanian mobile app for sign language translation developed by Lukas Gužauskas, a graduate of the Kaunas University of Technology (KTU) Faculty of Informatics, and Laurynas Bičkauskas, a master’s student at KTU, aims to fill this gap. The app is also an alternative way of text typing.
According to the World Health Organisation, over 5% of the world’s population – around 466 million people – have a hearing disability. It is estimated that by 2050 this number will reach more than 900 million.
A smartphone for an easier communication
The app created by the KTU students allows learning the international sign language alphabet. It has a list of sign language imagery that consists of 29 different letters of the international sign alphabet.
“We plan four basic functions available to the app user: a preview of the international sign language alphabet; a real-time alphabet letter prediction from camera view, recording text on the screen; and human language translation – from audio to on-screen text,” explains Gužauskas, a graduate of the Software Systems study programme.
While using the app, communication between a sign language user and a hearing person will no longer require a piece of paper. The smartphone camera scans the gestures of a person with a hearing disability shown in their native language, and the other person is speaking. The app delivers both ways in a textual format.
According to Gužauskas, the app has several settings. The first is the hand detection setting. The user can toggle between left- and right-handed usage. The other is speed and accuracy of the prediction.
According to the creators of the app, there are open source software based sign language translator versions in the market, however, all of them are designed for computer use. The aim of the team was to design the first app for a readily accessible device, a smartphone.
A challenge turned into an advantage
The communication with a person having a hearing disability can be challenging if one doesn’t know sign language. It is hard to understand for a hearing person because the sign language has a different sentence structure and grammar.
“Problems are common because hearing people often don’t understand what the deaf person writes and vice versa. The sign language cannot be translated literally. Because of the hearing impediment, it can become hard for a deaf person to write a grammatically correct sentence, and hearing readers can get the message wrong. I have noticed that live communication with the hearing can often be too difficult for a deaf person”, – says Gužauskas, who has the hearing disability and knows the problem first-hand.
The student says that deaf people usually have to use the services of the sign language interpreters, but they need to be booked in advance.
Mantas Lukoševičius, a researcher at the KTU Faculty of Informatics and the supervisor of the final project of the students, says that in the process, he had also encountered the communication barrier.
“They have turned a challenge into an advantage: knowing the problem well they created a tool for everyone. It’s great to work with students like Laurynas and Lukas. They went all the way from the idea to the final result. The students have identified and refined the problem they are solving and collected the necessary data – the images of the sign language alphabet. They have also done experiments with modern artificial intelligence methods that best recognise those images, and successfully transferred the trained neural network model into a smartphone”, says Lukoševicius.
The beta version of the Gesture Alphabet Translator app is now available on Google Play.
