Speech Generation From Hand Gesture Recognition For The Vocally Impaired

Project Title

Speech Generation From Hand Gesture Recognition For The Vocally Impaired

Project Area of Specialization Wearables and ImplantableProject Summary

Project Summary:

The ability to speak is a blessing from Allah but unfortunately, some people are deprived of this blessing. There are over 70 million people in the world who use manual communication to convey their meaning. Manual communication includes simultaneously employing hand gestures movement, the orientation of fingers, arms or body, and facial expression to convey the message. Manual communication is mostly employed through sign language.

Vocally impaired people face problems almost on a daily basis. Our project, the SG glove (Speech Generating glove) has been designed for the vocally impaired persons to help them communicate with other people having no knowledge of sign language. The SG glove is a cost-effective, easy to wear, portable glove that is worn on the right hand, with a battery time of more than one and a half hours. It converts Pakistani Sign Language (PSL) gestures into the Urdu language of the desired accent and gender tone. It recognizes more than thirty PSL gestures that are in daily use of the vocally impaired persons.

The hand gestures are sensed through nine flex sensors, which detect the bending of fingers and an accelerometer which detects the orientation of the hand.

The sensed data is received by the controller and then it is compared with a library of gestures. If the obtained values match the predefined values then gesture-related Urdu speech is obtained from the speakers.

The gesture-related Urdu speech words are stored in an SD card. The Urdu speech stored can be of either gender or any accent of Urdu language. These sounds are played through an MP3 player module.

Aiding the vocally impaired persons in bridging the communication gap with others is the main aim of the SG glove, and it has been designed after considering their everyday difficulties to make their lives easy and comfortable.

Project Objectives

Objective:

To develop Speech Generation from Hand Gesture Recognition for the Vocally Impaired persons which reduce the communication gap between vocally impaired people and the normal people. It provides a medium of communication for the vocally impaired people to communicate with normal people without any reluctance.

Project Implementation Method

Project Implementation Method:

The implementation of this project is that  when a vocally impaired person makes a gesture in Pakistan Sign Language (PSL) in order to talk to a normal person, the flex sensors bend which are placed on fingers metacarpophalangeal (MTP) and proximal interphalangeal (PIP) joints in order to get our desire data more precise, because they act as variable resistor (potentiometer). This bending of flex sensors generates analogue voltage signals to be received by the controller (Arduino Mega 2560).In the meantime, the orientation of hand is also detected by the accelerometer and then read by the controller. Then this data of flex sensors and accelerometer is processed in the controller which matches it with the desired pre-stored voices in MP3 format in micro SD Card which is placed in MP3 media player standalone sensor compatible with Arduino Mega 2560. This media player has 24-bit DAC which converts digital signals provided by the controller into voices of respective gestures input. As a result, speech is generated on speakers in the Urdu language. All the components of SG Glove are operating by a battery which makes our SG Glove portable, convenient and feasible.

Block Diagram:

Schematic :

Benefits of the Project

Benefits of the Project :

The proposed application of the SG gloves is to help the vocally impaired people in communicating with people who do not understand PSL. When our glove proves to be efficient, it can also be used for applications such as:

•    Language controlled Home or Industrial automation.

•    Virtual and Augmented Reality Gaming.

•    Addressing the mass audience using gestures.

•    Teaching for the vocally impaired persons.

Technical Details of Final Deliverable

Technical Details of Final Deliverable:

we are using the following hardware components:

Flex sensor:

we use a flex sensor to understanding the bending of the fingers. It's a Passive resistive device or bend sensor that measures the amount of deflection or bending. It uses as an analogue resistor. It works as a variable analogue voltage divider. It consists of a carbon resistive element within a thin flexible substrate.

Flat Resistance	25K Ohms
Resistance Tolerance	±30%
Bend Resistance Range	45K to 125K Ohms (depending on bend radius)
Power Rating	0.50 Watts continuous. 1 Watt Peak

Accelerometer:

We use the accelerometer (ADXL345) to understanding the orientation of the hand. The ADXL 345 is a 3 axis micro-electro-mechanical system (MEMS) based accelerometer and gyro on the same silicon die. This is used for recognizing all orientation of the hand. It gives a constant variable analogue voltage proportional to the acceleration applied in x y z directions.

ACCELEROMETER AT REST :

We still get the treading from accelerometer when it is at rest due to zero-g output (typically 3.35volt, depending upon the accelerometer model ).if a sensor is in a rest state on a horizontal plane then it measures 0g in X and Y axes whereas the Z axis will measure 1g,this Z-axis value is the output of the accelerometer.

MP3 Module and Speaker:

We use the MP3 module for generating speech output. It is quite small and low power MP3 module with a simplified output directly to the speaker up to 3W. It only requires few pins connections to Arduino and a speaker or any other with RX/TX capabilities to work. Looks really nice for my project.

It is compatible with sampling rates 8 kHz to 48 kHz and 24 -bit DAC output, support for dynamic range 90dB, SNR support 85dB. Its fully compatible with FAT16, FAT32 file system, maximum support 32GB TF card, 64M bytes NORFLASH and control variety of modes, I/O control mode, serial mode, AD control mode. It arranges audio voice by folder and supported up to 100 folders, every folder can carry up to 255 songs with adjustable volume is 30, 6 -level EQ adjustable.

Microcontroller (ATMega 2560) :

Arduino Mega is a microcontroller board based on the ATmega2560.It is a low cost and low power microcontroller contains 54 Digital I/O Pins including 14 PWM output, RX/TX and serial communication pins, and another 16 analogue pins with flash memory 256 KB and maximum clock speed 16 MHz quartz crystal and a reset button and other specification capable to handle large amount of data.

Battery :

Lithium-ion batteries are very popular nowadays because they are the most energetic rechargeable batteries available. They are much lighter in weight as compared to other rechargeable batteries of the same size the electrodes of a lithium-ion battery are made from lightweight lithium and carbon.

All the components of SG Glove are operating on a 7.2 Volts and 2.6 AH Lithium-ion battery which makes our SG Glove portable, convenient and feasible.

 Flat Resistance

 Resistance Tolerance

 Bend Resistance Range

 Power Rating

Final Deliverable of the Project Hardware SystemType of Industry Medical Technologies Wearables and ImplantablesSustainable Development Goals Good Health and Well-Being for PeopleRequired Resources

Elapsed time since start of the project	Milestone	Deliverable
Month 1	Litrature Review	Understand the complexity and flexibility of the Speech Generation (SG) Glove.
Month 2	Research On Plans	It Helped in understanding the essential requirements of the device.
Month 3	Research on Sign Languages	It helped to Learned Pakistan Sign Language
Month 4	Research of Component Availability	Helped to understand what kind of sensors would be used in the device.
Month 5	Flex Sensor testing	Observe and analyze their output readings and record their bending ranges.
Month 6	Acclerometer Testing	Observed and analyze its output readings and record its orientation ranges.
Month 7	MP3 Module testing with speaker	Stored voices in an MP3 file on SD card which is placed in Sthe D card module and then Generate the speech output on the speaker.
Month 8	Interfacing with Micro-controller	Interfaced Flex sensor, Accelerometer and MP3 Module with Arduino Mega and then analyze the output using recorded ranges.
Month 9	Circuitry testing	Helped in the configuration of a circuit perfectly that will generate speech output.
Month 10	Glove Assembly	Stitched the flex sensors on the finger of the glove and then Made a zippered pouch in which our circuit is placed. This box will be placed on our hand in order to make our SG Glove completely portable.
Month 11	Gesture Sampling	Start taking gestures samples and develop the coding according to Pakistan sign language.
Month 12	Final touches and Report Writing	Finalized our SG Glove physical appearance and starts making gestures with phrases in Urdu in the form of speech output by speakers and to make it completely portable connects our controller by a battery which provides it a long battery time.and made an FYP report of the SG glove.