Sign Language Recognition Using Image Processing

A gesture is a pattern that may be static, dynamic or both, and is a form of nonverbal communication in which bodily motions convey information. Communication is an important aspect when it comes to share or express information, feelings, and it brings people closer to each other with better underst

Project Title

Sign Language Recognition Using Image Processing

Project Area of Specialization

Artificial Intelligence

Project Summary

A gesture is a pattern that may be static, dynamic or both, and is a form of nonverbal communication in which bodily motions convey information. Communication is an important aspect when it comes to share or express information, feelings, and it brings people closer to each other with better understanding. When it comes to disabled persons for example deaf and dumb people, it becomes tougher for them to communicate using natural language. So, they use sign language to communicate with themselves and with the entire world. But normal people find it difficult to understand sign language as they do not have mostly any prior education or experience in this. Sign language is composed of visual gestures and signs, which are used by the deaf and mute for their talking. It is a well-structured code gesture where every sign has a specific meaning allotted to it. These signs are not only used for alphabets or numeric but also for common expressions also for example greetings and sentences There are 143 existing different sign languages all over the world, mainly American Sign Language (ASL), British Sign Language, French Sign Language, Japanese Sign Language, and Indian Sign Language (ISL). Every country has its own language, similarly, sign language is not a universal language and differs from country to country. There has been a lot of work already done on ASL recognition as it is a widely learned language all over the globe. ASL uses a single hand in the gesture representation and it is simple compared to ISL. ISL uses both hands for gesture representation and it is complex comparing to ASL. Because of this reason, there is less research and development in this field. This project goal is to take the simple step in connecting the social and communication bridge between regular people and disabled people with the help of Indian Sign Language. As our project only deals with alphabets and numeric in ISL, it can be extended to common expressions and also words which can be more effective for disabled and normal people in communication and understanding. As we live in a century where India is developing at a rapid pace in terms of digital and technological advances, this project could be one of the steppingstones where technology meets humanity and help the hearing impaired and mute community.

Project Objectives

Indian sign language (ISL) is one of the challenging topics as it is in the rudimentary stage of its development, unlike American Sign Languages (ASL). This project aims at the classification of Indian sign languages using machine learning models. There has been broad research on ASL and adequate data is available to analyze it. As India is a multi-diverse country, there are several regions and cultures which results in different variations of languages for communication. So, there are very limited standard data sets, which have variations and noises. ISL uses both hands to make gestures instead of one hand, unlike ASL. It leads to the occlusion of features and this is a major barrier to the lack of development in this field. This project aims at helping in the research of this field further by providing a data set of ISL. A data of sign language was created by us for alphabets and numeric. Later, the features will be extracted from the collected segmented data using image pre-processing and Bag of words model. Histograms are generated to map the alphabets with images. In the final step, these features will be fed to supervised models for classification.

Project Implementation Method

The algorithm implemented in this recognizes Indian Sign Language gestures taken from static pictures. The system comprises several steps which are Image collection, Image pre-processing (segmentation), Feature extraction, Classification. Bag of visual words (BoW) model has been implemented to classify the images. The idea of BoW is adapted from Natural language processing (NLP). In image processing, BoW model concept can be called a “histogram-based representation of independent features”. So, an image can be viewed as a document in order to depict any gesture using the BoW model. Likewise, it is important to describe “words” in images too. To accomplish this, the following three steps are normally included: feature description, and generation of codebooks (visual words). Using these codebooks, histograms can be generated for all the images. Further, the classification of images can be done using Support Vector Machine (SVM) model.

Benefits of the Project

ISL is a key for communication for deaf and dumb people in India. This paper gives a detailed implementation for Indian sign language recognition using the Bag of words model. In section 3, step-wise implementation has been discussed which are image collection, image pre-processing, feature extraction (using K-means clustering, visual words collection), and Classification. Finally, results were presented of the Bow model in section 4. Recognition for not only static images but also real-time recognition of gestures also developed. This project can also be extended for simple expressions and words in ISL including alphabets and numeric.

Technical Details of Final Deliverable

Using Bow, integrated with robust SURF feature descriptors, the model scored 99% accuracy. The confusion matrix for the model. All labels have been predicted correctly by the SVM except label 2. Real-time recognition prediction results. The precision, recall, f1 score was also 99.98%. But there could be slight biasing in the model prediction as the data set has many similar images without variations for example in light and skin tone. So, using a large and variety of images in the data set, this approach can be more robust for real-world applications.

Final Deliverable of the Project

Software System

Core Industry

IT

Other Industries

Core Technology

Artificial Intelligence(AI)

Other Technologies

Sustainable Development Goals

Good Health and Well-Being for People

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com