Sign language recognition and translation

Project Title

Sign language recognition and translation

Project Area of Specialization Artificial IntelligenceProject Summary

Since there are differently-abled people everywhere, it is crucial to make them feel included. Here we are specifically talking about people with hearing and speaking disabilities. such people have been using sign language quite a while ago and now sign language is widely used all around the world. Sign language is expressed by the movement of hands. Since most “normal” people do not understand this language, a huge communication barrier is created between them and people with disabilities. To make communication easy for such people, a system is required which recognizes those signs and translates them for those who are unable to understand.

So that's why we are making a project in which the Machine will translate for sign-language into text or spoken dialogue, then back into the text for deaf individuals, without needing a human interpreter.

The deaf has to simply signs in front of the camera and his sign video will be converted into frames and those frames will go through preprocessing of images. This is how we will take our input; this input and our data set will go through a convolution neural network and our pre-trained CNN model. the results will be shown after post-processing of images either in form of audio or video or text.  

The other way around, the reply from hearing individuals will be in form of text. The hearing individual will reply in microphone and speech/audio will be converted into text that will be shown on the screen of the deaf individual.  

we will design our own data set of alphabets by mixing different available data sets. We will use google colab for our code in contrast with google drive as our data repository. Along with this, we will use python for speech conversion into text with the contrast of speech recognition google API.  

Project Objectives

The only and basic object of our project is that we will be training the machine to detect the hand gestures and will translate them into words for the person who has no knowledge of sign language.

However, this communication works in both ways hence this machine will also have the ability to translate the spoken words into text for the impaired person to understand.

Most of the technology uses gloves, multiple cameras, microphones, Kinect, and other hardware devices. our object to reduce hardware as much as possible.

Our project will work for every individual and the device does not have to be reconfigured according to new users.

Project Implementation Method

We will use google collab free GPU for training a google drive to keep everything saved. We will perform object detection on custom in which is using tensor flow object detection API. The neural network we used in our project is a convolution neural network and the model we used for this project is a single shot detector SSD_MOBILENET_V2, Pre-trained on coco also known as common objects in context.

following is our implementation flow chart:

the first chunk is how we goanna retrieves our data from the database, we have used Google Drive as our data repository. the image will be taken from the repository and a convolution neural network theorem will be applied to it. Turn it will reach to our fully trained system. We will pre-train our system on coco. In the last of our first chunk, the features will be recognized in contrast to our frames of the input video.  

In the second chunk, the input will be taken from the camera, it will be in form of video and the system will automatically convert it into frames and will be preprocessed. Then those frames will be sent to the first chunk and their recognition and contrast will be done.  And the desired results will be Send toward post-processing of frames and output will be shown in form of video/audio/text.  

For the other way around, like a reply from speaking individual to deaf individual, we will use a different methodology. Speaking individual will give his reply in microphone and his voice will be converted into text and text will be shown to deaf individual and he can get the answer by reading it.

We will use python for this conversion. Along with python, we will use pyaudio library since it is used to receive audio input through the microphone and give output through the speaker. Basically, it helps to get our voice through the microphone. We are also using google speech recognition API in order to recognize the speech.

Benefits of the Project

Our project will help the deaf and dumb community of the world, our project will surely aid much towards effective and easy communication between normal and impaired individuals. Sign language translation is quite different from other types of language translation since it's structure is based on gestures 

After this project, deaf individuals will be able to communicate outside of their community and it assures 90% of effective communication with optimization. 

We are not using much hardware as it does not require sensors and multiple cameras at different angles, we will be using a single camera and the microphone that will reduce much of the hardware. Thus, it will be a time-saving system since it doesn't have to reconfigure itself or every other individual.  

Following our expected benefits:

• 90% Effective communication
• An optimized and efficient system for deaf
• Time saving System
•  

Technical Details of Final Deliverable

Our final product will be based more on software, and we will use less hardware. The person will simply sign in front of the camera and the camera will recognize the gestures and will translate them to hearing individuals. For this process, we would be using data set Google colab, Google Drive as our data repository, tensor flow API for object detection. We would be using a convolution neural network but the love the project will be a single shot detector SSD_MOBILENET_V2, Pre-trained on coco.  

Speech conversion we will use a microphone which will be used for input and the speech will be converted into text that'll be shown on the monitor screen of the deaf individual.  

Final Deliverable of the Project HW/SW integrated systemCore Industry ITOther IndustriesCore Technology Artificial Intelligence(AI)Other TechnologiesSustainable Development Goals Decent Work and Economic Growth, Reduced Inequality, Partnerships to achieve the GoalRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	26400
Raspberry Pi 4 Mode	Equipment	1	15000	15000
Raspberry Pi camera	Equipment	1	5500	5500
Connecting wires	Miscellaneous	20	10	200
Stationary	Miscellaneous	10	70	700
Printing	Miscellaneous	1000	5	5000