Awaaz

Project Title

Awaaz

Project Area of Specialization Artificial IntelligenceProject Summary

Spoken language is the medium of communication among the majority of the people, but perhaps over 5% of the people or 466 million people cannot use spoken language due to their speaking disability. Sign language comes to the aid of this community. Sign language recognition automation is an overlooked concept despite there being a large community of it. About 3.3 million Pakistanis are suffering from some kinds of disability in which 0.24 million are impaired of hearing/speech which approximates to 7.4% of overall disables. A very important point is that 55% of the total disabled lie in the age group from 5 -29 years. So, in order to facilitate this large social group, we are proposing an automated sign language recognition mobile app which will provide an opportunity for a speech-impaired person to communicate with the world without the need of an interpreter.

Deaf individuals have to rely on writing back and forth in English, for medical appointments, job interviews and orientations, business meetings, and legal situations. And, there were many terrible situations that arose from the inadequacies in the communication access for people who were deaf.

Currently, there is no application available for sign language publicly and communication with the deaf is dependent on an interpreter. Introducing sign language application publicly will replace the interpreter and it will fill the communication gap between a normal person and speech impaired person.

The application uses advanced deep learning tools to embed the classification model on the device hence creating no delay and latency issues. Open computer vision library is being used to detect hand from a frame and to preprocess the acquired frame. Deep convolutional neural network is used to train the model on 90 thousand images for accurate gestures prediction. Convolutional neural network has outperformed all the other image classification algorithms and it is being used by Facebook, Google, Pinterest due to its highly efficient results. Application after recognition shows textual and audio interpretation to the user.

The application is going to classify almost 85% of the results correctly.  The application can run offline without compromising accuracy. Sign language automation is an overlooked concept and introducing this application will result in decreasing dependency on an interpreter for communication

Project Objectives

• User can recognize gestures without the need of an interpreter
• The application will suggest words to deaf and provides auto-complete feature.
• The application will recognize gestures through live video and recognized gestures will be shown in the form of text.
• User can also listen to the audio interpretation of the recognized text 
• User will be able to upload already captured signs from gallery

Project Implementation Method

The software lifecycle is broken into cycles, each cycle working on a new generation of the product. The Rational Unified Process divides one development cycle into four consecutive phases

• Inception phase
• Elaboration phase
• Construction phase
• Transition phase

Each phase is concluded with a well-defined milestone—a point in time at which certain critical decisions must be made, and therefore key goals must have been achieved

The system is divided into three subsystems. First one is the presentation layer whose task is to capture signs through the front and back camera and send it to sign detection system. Secondly, it will show suggestions to the deaf which will help in reducing time and effort to write lengthy sentences. When Recognized gestures will be predicted it will be shown to the user in the form of text. The system will play audio interpretation as well. When the sign is captured it is sent to Sign Detection System it will preprocess the frame. The preprocessing includes the following steps:

• Detect the hand based on his skin colour segmentation
• Apply a binary mask on the frame
• Apply morphological operations

After preprocessing the frame is sent to classification model which will predict which sign the user has made and It will send the predictions to the presentation layer which will output the results on screen and audio interpretation will play accordingly.

Benefits of the Project

• The application will help in reducing dependency on an interpreter
• User can use the application anywhere at any time without any cost
• User can use the application even without the internet
• The application will help to achieve direct communication with a deaf
• There would be no need to take interpreter along with you

Technical Details of Final Deliverable

Awaaz is an intelligent sign language recognizer application that will bridge the communication gap between a speech impaired and a normal user. The application can be used by the deaf and even by the normal people. User will have two options to predict gestures. User can upload signs from the gallery and the application will play an audio and textual interpretation of the signs to the user. Secondly, User can use live video module to talk with a deaf. When live video module will be used camera will turn on and deaf will make signs in front of the camera. Deaf will make signs and application will recognize gestures and textual output will be shown to the user on-screen. When a sign is made application will detect where the user’s hand is based on the skin colour segmentation. Afterwards a binary maks will be applied on the frame which will turn detected hand colour to white and rest of the region to black. The application will then classify the masked frame through the model. The model will return predictions and highest predicted text will show on Screen. Suggestions will be shown on screen and if the deaf also wanted to say the suggested word deaf will make a sign for space for two times and suggested word will be concatenated to the text. If deaf is done with its talk he/she will make a thumbs-up sign and the application will play an audio interpretation of the whole predicted text.    

Final Deliverable of the Project Software SystemCore Industry ITOther Industries Education , Others Core Technology Artificial Intelligence(AI)Other Technologies OthersSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	74900
Nvidia Geforce Graphic Card 1080	Equipment	1	65000	65000
Platform and device compatibility testing	Miscellaneous	1	6000	6000
Deep learning course udemy	Miscellaneous	1	2100	2100
Convolutional Neural Network Udemy	Miscellaneous	1	1800	1800