Real Time Sign Language Interpreter

Project Title

Real Time Sign Language Interpreter

Project Area of Specialization Artificial IntelligenceProject Summary

Deaf people around the world communicate using sign language. It is a visual language that uses a system of manual, facial and body movements as the means of communication. Sign language is not a universal language; different sign languages are used in different countries. Hundreds of sign languages are used around the world, for instance, Japanese Sign Language, British Sign Language (BSL), Spanish Sign Language, Turkish   Sign   Language. In the same way, Pakistan also has its own sign language i.e. Pakistan Sign Language or PSL.

Since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using neural networks for finger spelling based Pakistani sign language. It will be a mobile application that will, with the use of AI, translate PSL to text/speech and similarly text/speech to PSL. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures.

Project Objectives

Since every one cannot understand sign language, it becomes difficult for deaf/mute people to communicate with the hearing community. This application focuses on solving this problem by translating PSL to text/speech.

With enough research already been made in this field, it is only the implementation which needs to be done so that the present resources could be put to use in order to create something that solves this problem. Some applications have been made but none of them translates PSL. This is where this application comes into play. The sign language interpreter will be a mobile application that will, with the use of AI, translate PSL to text/speech and similarly text/speech to PSL. In this way, the deaf person will be able to communicate with the hearing person and the hearing person will be able to communicate with the deaf.

Project Implementation Method

This project aims at translating sign language to text/speech and vice versa. This will be done in real time by recording the live video of the hands of the signing person. The video frames will be passed on to the neural network which will classify each sign and map them to the alphabets or the words they represent. This will produce an output which will then be passed on to RNN in order to create a sentence out of the words.

Benefits of the Project

The world around us is built to accommodate the hearing population residing on it. However, people from the deaf and hard of hearing community are usually not properly accommodated especially in a country like Pakistan. There are approximately 10 million hearing impaired citizens in Pakistan. With such a huge number, Pakistan still lacks in providing accessibility to these ten million people. Due to this there is a general lack of opportunities for this community in terms of education and jobs.

People from the deaf and hard of hearing community have difficulty in performing everyday tasks that should not be a hindrance but they are, because of the communication gap between them and hearing people. For e.g. if a deaf person goes out for grocery and requires help from the staff, there is a high chance that none of the workers there would know any PSL. This is where this project would be helpful.

Technical Details of Final Deliverable

The product will be used on Android mobile phones. Since the application is made to run on only Android OS, it won’t run on IOS. The in-built mobile device camera will be used to capture video. A smart phone with good quality camera is recommended so that clear video is captured.

The application will, record live feed of the hands of the signing person and with the use of AI, translate PSL to text/speech and similarly text/speech to PSL. In this way, the deaf person will be able to communicate with the hearing person and the hearing person will be able to communicate with the deaf.

Final Deliverable of the Project Software SystemCore Industry ITOther IndustriesCore Technology Artificial Intelligence(AI)Other TechnologiesSustainable Development Goals Reduced InequalityRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	50000
Camera (training purpose)	Equipment	1	25000	25000
GPU (training purpose)	Equipment	1	15000	15000
Mic (training purpose)	Equipment	1	5000	5000
Reports + Standee + Poster	Miscellaneous	1	5000	5000