hand gesture to speech conversion for dumb people

It is very difficult for dumb (speechless) people to interact with normal people because most people are not able to understand sign language. We are developing a smart glove that can translate hand gestures into speech. We are integrating flex sensors and Mpu6050 (gyroscope + accelerometer) wi

Project Title

hand gesture to speech conversion for dumb people

Project Area of Specialization

Wearables and Implantable

Project Summary

It is very difficult for dumb (speechless) people to interact with normal people because most people are not able to understand sign language. We are developing a smart glove that can translate hand gestures into speech. We are integrating flex sensors and Mpu6050 (gyroscope + accelerometer) with raspberry pi. Data of hand gestures are collected from these sensors and then this data is processed. After preprocessing different machine learning algorithms (SVM, KNN, DTC, ANN) are used to train the model and finally compare all these algorithms to find out which is most suitable, and then the trained model is tested in real-time to view its accuracy and sustainability. It is a multi-purpose glove that finds its applications in controlling digital devices, robotics, Virtual Reality, and many others.

Project Objectives

• Sign Language to Voice conversion
• Bridge b/w normal and special individuals
• Pakistani Sign Language gestures

Project Implementation Method

Project Implementation Method:

Working methodology of our project is described as:

• Gesture recording
• Data pre-processing
• Classification of gestures (using ML algorithms)
• Text to speech conversion

1. Gesture recording:

      Gestures are captured using flex sensors and MPU 6050 (gyroscope + accelerometer). The flex sensor works as a voltage divider when connected with a resistor. It tells us about the bend of fingers in the form of analog values. So, we have to convert it to digital form so that our micro-controller can understand it. So, an analog to digital converter is used to make this information valuable to a microcontroller. Moreover, MPU-6050 will be used for the orientation of the hand in space. Thus, the values of MPU-6050 will be conveyed to the microcontroller. Thus, the flex sensor will give the values but the gyroscope will determine its orientation in space and gesture will be recorded accordingly.

2. Data pre-processing:

      Null values and stray values are removed from the dataset to make the data clean and valuable. Moreover, data is split into training and testing with a ratio of 70:30.

3. Classification of gestures (using ML algorithms):

      After getting cleaned data different classifiers are implemented. Decision Tree Classifier gives an accuracy of 95.07%. K Nearest Neighbor gives an accuracy of 96.23%. While Artificial Neural Network gives an accuracy of 98.112%. So, the classifier returning the highest accuracy is implemented.

4. Text to speech conversion:

      When a gesture is recognized, its respective speech is its output in the form of text. Now, this text is converted into speech using Python Text to Speech (pyttsx) library. This creates a .wav extension file that can be played by any speaker. So, a speaker is attached with Bluetooth to play voice files of gestures. So, when a gesture is recognized its voice file is automatically played. In this way, a gesture is converted into listenable sound.

Benefits of the Project

Smart glove converts sign language to speech 

It can be used in Virtual reality applications.

It can control different electronic devices with gestures.

It can manipulate a robotic arm

Technical Details of Final Deliverable

Attributes

Input

Output

No. of gestures recorded.

No of samples

Accuracy

Components

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Medical

Other Industries

IT , Health

Core Technology

Wearables and Implantables

Other Technologies

Artificial Intelligence(AI), Augmented & Virtual Reality, Robotics

Sustainable Development Goals

No Poverty, Good Health and Well-Being for People, Reduced Inequality

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com