Hand Gesture Vocalizer for Mute People

Project Title

Hand Gesture Vocalizer for Mute People

Project Area of Specialization RoboticsProject Summary

Those with speech impairment due to autism or cerebral palsy, use sign language for communication, and – understandably – find a lot of difficulties in conveying their messages. This difficulty equally holds at the receiver’s side for trusted understanding and interpretation. To overcome their communication barrier, an Arduino UNO based augmentative and alternative communication (AAC) system is proposed, which is capable of translating hand-gestures into text, which are later vocalized. The system requires the end-users to be familiar with hand gestures for their essential needs a priori; they are normally taught by speech-language pathologists. The core component of the system is a flex sensor, whose resistance increases with its bend. We stitch such sensors on a hand glove, and each bend of a finger – due to a different resistance – encodes a unique gesture; thereby allowing multiple gestures to be communicated by each finger. These coded messages are wirelessly transmitted using a Bluetooth transmitter-receiver pair, where the latter is connected to another Arduino UNO module, responsible for correct decoding. Once decoded, the message is displayed on an LCD, and subsequently vocalized by means of text-to-speech converter.

The proposed system has a potential to be shaped into a telephone set for persons with speech impairments, where hand-gestures will be converted into text and voice, and the voice, on return, will be converted into visual gestures, displayed on a screen in real-time. This will require natural language processing, which is something we plan to do in future.

Project Objectives

The primary objective of the proposed system is to assist persons with speech disability and impairments, in conveying their messages to the people around them. The specific phase-wise milestones of the project are as follows:

1. Designing a wearable flex-glove
2. Programming the coding of varying resistances of flex-sensor
3. Wirelessly transmitting the encoded message using Bluetooth module
4. Convert the decoded message into text and speech  

Project Implementation Method

The implementation of the project involves software and hardware components, and is based on a suitable microcontroller. For the first objective, flex sensors will be stitched on a glove, whose resistance, for each bend, will be digitized and conveyed to a microcontroller. The latter, each at the transmitter and the receiver end, will be responsible for encoding and decoding the message behind each bend of the finger respectively, where the transmitter and receiver pair will communicate using a Bluetooth module. Once correctly decoded by the receiver, the message will be displayed on a screen and later vocalized for conveying the message.

Benefits of the Project

Societal contributions

1. An autonomous system capable of giving voice to mute people, thereby serving a neglected segment of our society.
2. Requires minimal human intervention for communication

Literary contributions

1. The algorithm for encoding and decoding has a potential to be published.
2. In future, an updated version of the same will incorporate natural language processing, which will be a significant contribution for the scientific and research community.

Commercial benefits

1. The system to be developed, if successfully transformed into a product, will be submitted for patent. We are confident that a full-duplex system will have a potential to be commercialized.

Technical Details of Final Deliverable

1. Easily scalable and modifiable for the inclusion of new gestures
2. Off-the-shelf components make debugging and replacing of faulty components convenient and cheap.
3. Communication range, in the first prototype will be ten meters, which may be increased by means of Wi-Fi and/or GSM connectivity.
4. 45-125K ohm resistance of flex sensors
5. 5 volt and 40 Amp Arduino UNO will make the system energy efficienct.

Final Deliverable of the Project Hardware SystemType of Industry Others , Health Technologies RoboticsSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	46880
Flex sensor	Equipment	10	1750	17500
Arduino UNO	Equipment	4	800	3200
Emic-2 module	Equipment	2	10000	20000
Bluetooth HC-05 module	Equipment	4	570	2280
Gloves	Equipment	2	300	600
Jumper wires	Equipment	100	10	1000
Speaker	Equipment	2	150	300
2*16 LCD	Equipment	2	250	500
Report printing	Miscellaneous	10	150	1500