Realization and Implementation of Communication for hearing impaired and inarticulate people
Natural language processing is one of the most growing fields of research. It deals with human-computer interaction and natural languages. Humans share ideas and thoughts with people around them using speech and hearing abilities. But this is not the case for hearing impaired and inarticulate people
2025-06-28 16:28:55 - Adil Khan
Realization and Implementation of Communication for hearing impaired and inarticulate people
Project Area of Specialization Computer ScienceProject SummaryNatural language processing is one of the most growing fields of research. It deals with human-computer interaction and natural languages. Humans share ideas and thoughts with people around them using speech and hearing abilities. But this is not the case for hearing impaired and inarticulate people. Through sign recognition, communication is possible with hearing impaired and inarticulate people. This project aims to develop a mobile application-based system for recognizing sign language and converting it into text and speech, which provides smooth communication between the deaf-mute community and normal people, thereby reducing the communication barrier between them. Our proposed system will consist of a mobile application that would be used for communication. The project will mainly consist of two parts, i.e., sign-to-speech conversion and mobile application development. In sign-to-speech conversion, the sign will be recognized using a mobile camera with the help of computer vision, image processing and machine learning algorithms. The recognized sign will be converted into text, and text will be used to generate speech. The whole system will be integrated with easy to use mobile application. With the incorporation of the android mobile application, this system will ease people with disabilities to communicate with non-disabled people and reduce the communication barrier between them.
Project ObjectivesNatural Language Processing (NLP) is an emerging field that helps in conveying information and meaning with semantic cues such as words, signs, or images. Sign Language is a very convenient way for hearing impaired and inarticulate people. But for other world, communication with hearing impaired and inarticulate people is difficult because a normal person cannot understand their sign language in normal circumstances. This difference in society can be minimized if we can program a machine in such a way that it translates sign language into text or speech. According to a survey, there are more than 10 million people in Pakistan who cannot express their thoughts and feelings in speech. Therefore, we proposed a project that utilizes computer vision and image processing techniques to develop a system that can convert sign language into textual and audio form with a mobile application interface, thereby, reducing the communication barrier between specially-abled and non-disabled people. This project aims to meet UN’s SDGs 8.5, 10.2 and 10.3. It aims to reduce inequalities and communication barriers in society. The second purpose of this project is to increase the productive employment of all persons including persons with disabilities.
Project Implementation MethodFor the implementation of our project, we have to collect the video dataset based on sign gestures (from specially-abled people) to train the machine learning model. Sign language depends on continuous gestures of hands movement and facial expressions rather than static images. So we have to collect data in the form of videos based on daily life activities. This video dataset will be fed to a machine learning model which will be used to classify the sign gestures. After successful results and satisfactory accuracy of the model, the whole system will be integrated with a mobile application for easy useability. The application will display the predicted text from sign gesture and generate speech as the final output.
Benefits of the ProjectWe are part of society and we should try to solve problems in our societies. Communication is a basic right of each person in society. But this is not the case for hearing-impaired and inarticulate people. There are more than 10 million people in Pakistan who cannot communicate or participate in many social, political and professional events due to their inability to speak and hear. There is a lack of accessibility for these people in nearly every field of life such as markets, job interviews, and day-to-day communication. This project utilized computer vision and image processing techniques to develop a system that can convert sign language into textual and audio form with a mobile application interface, thereby, reducing the communication barrier between specially-abled and non-disabled people in the society. It is a reliable alternate mode of communication that provides the following benefits:
- It does not need a voice and is beneficial for individuals suffering from hearing and speech disabilities to improve their quality of life.
- The system will translate sign gestures into words and sentences which are important for communication.
- This system does not require any hardware and is based on a mobile application that is easy to use and available anywhere. The user does not need to carry any additional equipment with him in order to communicate.
This system will bridge the communication barrier in society and will open doors to employment and opportunities for specially-able people. By bridging the communication gap, the life experience of the deaf-mute will be enhanced by allowing them to express themselves and being understood better.
Technical Details of Final DeliverableWe will deliver a software-based mobile application that will be easy to use and have a friendly user interface. This mobile application will provide facilities of:
- Recording a sign gesture video of deaf-mute people.
- Uploading existing sign gesture video.
- Prediction of the sign in the form of Text and Speech.
| Item Name | Type | No. of Units | Per Unit Cost (in Rs) | Total (in Rs) |
|---|---|---|---|---|
| Total in (Rs) | 55400 | |||
| Google Colab Pro Tool | Equipment | 3 | 1800 | 5400 |
| Cloud Services Tool | Equipment | 1 | 25000 | 25000 |
| Video Recording Equipment | Equipment | 3 | 5000 | 15000 |
| Others | Miscellaneous | 5 | 2000 | 10000 |