Sign Language Communicator

According to the World Health Organization(WHO),  there are over 360 million people around the globe that are suffering from disabling hearing loss. In those 360 million people some are completely deaf that basically have very little or no functional hearing at all and the other are ca

Project Title

Sign Language Communicator

Project Area of Specialization

Augmented and Virtual Reality

Project Summary

According to the World Health Organization(WHO),  there are over 360 million people around the globe that are suffering from disabling hearing loss. In those 360 million people some are completely deaf that basically have very little or no functional hearing at all and the other are called “hard of hearing” who have mild-to-moderate hearing loss. Deaf people use sign language to communicate whereas the hard of hearing can use sign language and spoken a language with aid too. Deaf communicate in sign language as their first language. Abled people perhaps have little or no knowledge of sign language at all. Deaf people have difficulty in correspondence on a daily basis. A sign language interpreter may be used but they are expensive to have at hand an interpreter all the time and inconvenient too. A solution to this problem is needed that is economical and easy to use. Hence, our final year project proposes a solution to this problem that is easy to use and have to. The solution wished-for is an android application works both ways. First as a listener i.e. it detects the signs performed by the deaf person using Kinect and translate them to native spoken language in form of text on an android tablet screen then secondly  as a speaker i.e. it takes the text in natively spoken language typed by the abled person through keyboard  and converts into a series of sign which is implemented on a 3D avatar in augmented reality on the screen as in projected on that person as if that person is performing them in reality.

---

Project Objectives

Objectives of this Final year Project is to be the voice for the deaf community who face many challenges in communicating on a daily basis. The objectives of the application are:

• Using Kinect to detect the signs performed by the deaf person.
• Translating those signs into text in the native language of the abled person on the tablet,
• The abled person typing an answer to the former query on the tablet’s keyboard.
• The texted answer is then translated into sign language.
• The series of sign gestures, motions are being mapped on the AR 3D avatar.

Keeping in mind that every human being in all normal cases has the same hand shapes consisting of 4 fingers and a thumb. This final year project aims at creating a real-time system that recognizes the meaningful shapes made by using hands.

---

Project Implementation Method

Kinect will be used to detect the deaf persons' action. Kinect, in turn, would be connected to the Android tablet. On that Android tablet, the Kinect would send the detected signs to the Sign Language Communicator application. This application is made in Java using the Android SDK. The Sign language communicator application will translate the signs into words of the native language. This translation is done using a trained model for sign language recognition. As soon as the recognition is done, the text translation of the signs on the applications user interface will be displayed to the abled person. Using the same interface the abled user can now type the sentence that needs to be translated. The Sign language communicator application now takes the text as input, translates it into a series of hand gestures or signs. This translation is also done using a trained model. The signs produced are then implemented on the 3D avatar which is made using Blender.

---

Benefits of the Project

As there are over 360 million people in the world that are disabled due to hearing loss. The only way to communicate with them is through sign language. This creates a communication barrier between the deaf and abled community. To bridge the gap this application will help an abled person understand what the deaf are trying to say and the deaf person to understand what the abled are saying.   As the application is easy to use so a layperson can also use it. It’s economical as has less complex hardware. This application so helps deaf to make an appointment with doctors, get things done that are otherwise hard for them to do themselves.

---

Technical Details of Final Deliverable

The final product will be an android application that will run on supporting platforms. In Listener mode, the Kinect will be connected to the Android platform that will detect the deaf person’s action and the android device will translate them into text displayed on the screen. In speakers mode, the person would type text and then that would be translated on the android device into a series of signs that would be implemented on a 3D avatar.

---

Final Deliverable of the Project

Software System

Type of Industry

Others , Health

Technologies

Augmented & Virtual Reality, Others

Sustainable Development Goals

Good Health and Well-Being for People

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com