Vision based Pakistan Sign Language to Text Translation

Project Title

Vision based Pakistan Sign Language to Text Translation

Project Area of Specialization Artificial IntelligenceProject Summary

Vision base Sign Language to Text Conversion is one of the potential applications of computer vision through deep learning. Communication is deeply intertwined with human existence. In personal life, we communicate to deal with certain concerns and problems of daily life. In professional life also, it is communication that helps us to build healthy relations with others. Unfortunately, people exist among us with certain disabilities with whom we face problems while communicating or are completely unable to communicate. One of them are hearing impaired people and Sign language is the most natural and expressive way for them to communicate with other people but normal people never try to learn sign language for interaction with deaf people which leads to the isolation and segregation of deaf people. In order to get rid of this communication barrier, there is a need of such reliable system that can translate sign language to regular text. The sign language on which this project is based-upon is Pakistan Sign Language (PSL) which is the primary language of deaf people in Pakistan. There are two primary approaches for hand gesture/sign recognition. The first one is sensor-based approach and the second one is vision-based approach. The sensor-based approach includes the measurement of electric signals, resistance etc. for recognition of hand gesture It is most accurate, but it is not user friendly, because gloves with various sensors affixed on it, are put on by person (normally by a hearing-impaired person) and hence it may make user annoy and may increase the overall development cost of the system. On the other hand, vision-based approach uses cameras in order to detect the hand gesture without attaching any sensing device to the human body, hence it is user friendly and more economical. The main aim of our project is to develop a vision-based system which will recognize the hand gestures based on Pakistan Sign Language (PSL) and generate the respective text in both English and Urdu language. So, the focus is placed on designing the system which will accurately detect the hand gestures in different lighting conditions, perspective and on the hands of different age groups

Project Objectives

The main objective of the proposed system is to recognize the dynamic hand gesture through live feed and as well as through pre-recorded videos. An image classifier based on deep neural network is proposed to classify different gestures by continuously tracking the hand movement form video frames and generate the text against the respective gesture and then embedding this classifier in mobile application. The proposed system can be evaluated for the effect of correctly classifying the gestures in different operation conditions. Our objective is to increase the accuracy of vision-based approach and motivate people to work more and more on Pakistan Sign Language (PSL).

Project Implementation Method

Vision-based hand gesture recognition is not an easy task. It requires considering many factors such as different hand types, different abduction phenomenon, different lighting conditions, different perspective etc. of the image on which the deep neural network will be trained. Using different image processing technique for this purpose can be a good solution to increase the accuracy of the system. Also, the speed of such systems is very important because they must recognize the hand gestures quickly and accurately. The basis of this system is the deep learning neural network which learn the abstract features of different hand gestures to recognize them in real world. There are many methods which allow the detecting of the individual hand gesture elements. They are based on both vector operations and pattern classification through image filtering in complex space or through image processing in spatial-frequency domain. Our system is using different image processing techniques and deep learning algorithm to detect and classify the hand gestures in any possible condition real-time and, we are using iterative development using waterfall process model in order to develop our system. The purpose of working iteratively is to allow more flexibility for changes.

Benefits of the Project

This system uses the advanced deep learning technology in order to recognize hand gestures in real-time. This system if installed at public places provide deaf-dumb people an efficacious means of expressing themselves and eradicates the need of sign language interpreter. This system can be installed at different transportation facilities like airports, railway stations, bus stations, police stations, hospitals, courts, government offices, schools, universities, and shopping complexes. Further the system’s functional code can be used to make a browser widget which will generate captions for every video played involving sign language.

Technical Details of Final Deliverable

The intention of the project is to devise a mobile application that will capture hand gestures from mobile camera in real-time as well as from the pre-recorded videos and generate captions against each captured gesture both in English and Urdu Language.

Final Deliverable of the Project Software SystemCore Industry ITOther Industries Education , Others Core Technology Artificial Intelligence(AI)Other Technologies OthersSustainable Development Goals Industry, Innovation and Infrastructure, Reduced InequalityRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	80000
Cloud computation services	Equipment	1	29500	29500
Cell phone	Equipment	1	30500	30500
Internet device+services	Equipment	1	10000	10000
Stationary, travelling etc.	Miscellaneous	1	10000	10000