DD SPEAK-TO (DUMB & DEAF SPEAKING SYSTEM)

Project Title

DD SPEAK-TO (DUMB & DEAF SPEAKING SYSTEM)

Project Area of Specialization Internet of ThingsProject Summary

Deaf-Dumb people used Sign Language for communication. It is essential to get dumb and deaf people involved and connected to each other. Dumb and Deaf Speaking System will be very helpful to the mute people for conveying their thoughts to others and understand others in the form of gestures. First, Dumb and Deaf people gestures are captured by kinetic camera and recognize their gestures by Gesture Recognition method and give output in the form of text or sound. After receiving the message, normal person will answer in the form of text and sound which is also converted into gestures by using 3D model which is displaying on the screen. This system also includes a text to speech conversion block that interprets the matched gestures

Project Objectives

• Efficient communication between Dumb and Deaf People and normal.  
•  Assign language involves different gestures and hand movement, improves small skills of different people to understand the thoughts of mute people.
•  A mobile application can be built for broadcasting communication.
• To include real time application 
• The image noise is reduced by using filters 

Project Implementation Method

 Methodology: 
5.1  Gesture to Text Conversion:  
First, gestures are captured by using a kinetic camera. A kinetic camera is used for capturing the gestures of mute people in the form of video. Then this video is converted into different frames. Each frame has an image and apply image processing on it. Image processing include remove background, repositioning and rescaling and pointing edges. After image processing gesture are recognized by Gestures Recognition method which involves different algorithms.

 Gesture Recognition Algorithms: 
a) 3D Model & Skeletal Model: 
 This is an algorithm that is used to create a 3D or three-dimensional model which detects an object and its position within a certain radius of the camera to identify the gesture portrayed. Remember that gesture can only be translated into text if a gesture is found in database.  
The skeletal model is a spatial model that has an algorithm which uses a simpler set of parameters such as joint angles and segment lengths. It can be analyzed by using their position, orientation, and relationship with other joints and segments. 
Now it searches for matching messages for the set of values. Once it is found in database memory, this message is retrieved and convert into phrases. Then gestures are converted into the text messages and also produce sound with the help of using speaker. Scikit-learn will be used to understand and analyze the meaning of gesture and result the output into text. 
b) Text to Speech Conversion: 
Text to speech conversion is often called TTS. Festival (the Festival Speech Synthesis System) is a framework for TTS. It builds speech synthesis systems with various example modules and offers a full text to speech system. We will use it through the Python wrapper Pyfestival. In order to keep our focus on gesture recognition and to reduce development time, we will use this existing library. In our project, Pyfestival will be used to convert text into speech. The text input for this tool will be obtained from the gesture interpretation. 

5.2  Speech or Text into Gestures:

This voice or text is again converted into gestures by creating 3D model which indicates the motion of mute people displayed on the screen

a) Speech to Text Conversion (STT): 

 This system focusses on real-time recognition of spoken language. Pypi is a Python library for speech recognition that supports several speech recognition engines and APIs including CMU Sphinx

b) Text to Pakistan Sign Language (PSL): 

 The text value retrieved from spoken English will be matched to the corresponding PSL model. Then the PSL posturer gesture will be passed to the next stage to be simulated on the computer screen.

Now we have a fully functional smart speaking system to help mute people communicate with normal people. All the conversation is capture by making a video using an Android app and then later it will be broadcast for conveying message to other dumb people in the world. 
 

Benefits of the Project

 Outcomes/ anticipated results  

• Facilitate the Dumb and Deaf people for conveying message. Better for two-way communication.
• Will be very effective in case of any emergency.
• Improves the learning skills of people.
• Friendly communication environment for conversation with mute.
• Develop a Mobile App for broadcasting their message over the world. 

Technical Details of Final Deliverable

Tools And Techniques:

• unity 3D
• Android
• Image Processing
• Text Recognition
• Voice Recognition
• Video processing
• Database

Final Deliverable of the Project HW/SW integrated systemType of Industry Education , Medical , Media , Telecommunication Technologies Internet of Things (IoT), Blockchain, OthersSustainable Development Goals Good Health and Well-Being for People, Industry, Innovation and Infrastructure, Reduced InequalityRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	35800
Kinetic Camera	Equipment	1	25000	25000
Book For Dumb And Deaf	Miscellaneous	1	1000	1000
15.6	Equipment	1	2000	2000
Printing Cost	Miscellaneous	1	1000	1000
XBOX 360 LT3 with Wired Controller and GTA V	Equipment	1	6800	6800