Sign Language Detection and Translation through Deep Learning

Project Title

Sign Language Detection and Translation through Deep Learning

Project Area of Specialization Artificial IntelligenceProject Summary

A language is a mode of communication among human beings, we use written or spoken language to communicate with others, however for the people who don't have the ability to communicate with either mode sign language is the only option.  So, this gap is filled by Sign language which is the basic communication method between deaf and normal people. Signly is a  project to conquer any hindrance between an ordinary person and a deaf individual. The main goal of this project is to develop a  sign language detection system that can translate the sign language into the meaningful text to ease the daily tasks of mute people so that disabled people are not left behind.   The purpose is to provide a space for effective communication methods between the deaf community and normal people. The project uses a visual-based approach (unlike the traditional Data-glove approach) to minimize the equipment necessities and costs among the deaf community. The deliverable would be an android application that can be easily used by deaf people to communicate with the normal person to perform their routine activities.

A language is a mode of communication among human beings, we use written or spoken language to communicate with others, however for the people who don't have the ability to communicate with either mode sign language is the only option.  So, this gap is filled by Sign language which is the basic communication method between deaf and normal people. Signly is a  project to conquer any hindrance between an ordinary person and a deaf individual. The main goal of this project is to develop a  sign language detection system that can translate the sign language into the meaningful text to ease the daily tasks of mute people so that disabled people are not left behind.  

The purpose is to provide a space for effective communication methods between the deaf community and normal people. The project uses a visual-based approach (unlike the traditional Data-glove approach) to minimize the equipment necessities and costs among the deaf community. The deliverable would be an android application that can be easily used by deaf people to communicate with the normal person to perform their routine activities.

  Project Objectives

The main objective of this project is to design a system that can fill up the gap between the normal person and the impaired people and assist them to communicate with normal people in a comparatively better fashion.  This project objective:  i. Developing a sign/gesture recognizing system that can recognize sign gestures of deaf and mute people of American Sign  Language (ASL)  ii. The sign language will then be converted into text with the functionality of translation and prediction of the correct meaningful sentence.   iii. To work on better accuracy and performance of the system.

The main objective of this project is to design a system that can fill up the gap between the normal person and the impaired people and assist them to communicate with normal people in a comparatively better fashion. 

This project objective: 

i. Developing a sign/gesture recognizing system that can recognize sign gestures of deaf and mute people of American Sign  Language (ASL) 

ii. The sign language will then be converted into text with the functionality of translation and prediction of the correct meaningful sentence.  

iii. To work on better accuracy and performance of the system.

 Project Implementation Method

First of all, we explore the previous work that is already done in the sign language by the people so, we got the idea of that how much work and implementation has been done by the other people. And what were the loop wholes were left by the people. What type of dataset do they use? Which model do they use? What are the library and environment required to implement? What are the use cases of our project? After that the previous work we found, we will try to implement that work and run the previous codes that we have found in this way we will get to know how the actual things are working. Secondly, we will find the most generic words and sentences that deaf people use in their daily life and then we will make the dataset of that words and sentences then we do the data pre-processing, model training, and data augmentation. On our dataset which model is performing well, and hyperparameter tuning will apply for getting the best model. Thirdly, we will build the small model and then we will integrate that model with the mobile application simply we build the API of the model, our main focus is to build the minimum viable product so that we can involve the user, and then he will help users to make the more use cases. Along with that, we will implement the basic UI of the application. Till here all the basic implementation will be done so now own words we will work to improve the model, which means we will follow the iterative model. So, we build the sprints of our implementation and iteratively we update the application in this way we try to implement the project.

First of all, we explore the previous work that is already done in the sign language by the people so, we got the idea of that how much work and implementation has been done by the other people. And what were the loop wholes were left by the people. What type of dataset do they use? Which model do they use? What are the library and environment required to implement? What are the use cases of our project? After that the previous work we found, we will try to implement that work and run the previous codes that we have found in this way we will get to know how the actual things are working. Secondly, we will find the most generic words and sentences that deaf people use in their daily life and then we will make the dataset of that words and sentences then we do the data pre-processing, model training, and data augmentation. On our dataset which model is performing well, and hyperparameter tuning will apply for getting the best model. Thirdly, we will build the small model and then we will integrate that model with the mobile application simply we build the API of the model, our main focus is to build the minimum viable product so that we can involve the user, and then he will help users to make the more use cases. Along with that, we will implement the basic UI of the application. Till here all the basic implementation will be done so now own words we will work to improve the model, which means we will follow the iterative model. So, we build the sprints of our implementation and iteratively we update the application in this way we try to implement the project.

    Benefits of the Project

There is a gap or we can say the barriers between the people who can speak and the people that are unable to speak (deaf people). And we are trying to remove this barrier so that both people can communicate with each other easily, moreover with the help of this project, deaf people can convey their message to any people like if deaf people have seen any crime and he want to tell/inform to someone else so this project will help our moreover if the deaf people want to teach the students then this project will help a lot. And if the deaf person wants to tell his disease to the doctor then he can convey his message easily. and

Technical Details of Final Deliverable

1. Sign Language Translation Models   a. Static Signs  i. Deep Learning Model For classification of different signs with no motion using MediaPipe + ML algo (to be  decided) / CNN / Yolo .  ii. Integrating It to app through TensorFlow lite Framework and MediaPipe’s implementation or make an API using  Python and deploy it.  b. Action Recognition  i. Deep Learning Model for action (signs with motion) recognition using MediaPipe + ML algo (to be decided). ii. Integrating It to app through TensorFlow lite Framework or make an API using Python and deploy it. 2. Text Processing Model  a. Raw text generated from translation models are converted into meaningful sentence.  b. Most probably we will use Recurrent Neural Network for text processing.  3. Mobile Application   a. Login/ Sign Up for each individual using the app to keep track of all translations.  b. Camera Tab Provide real-time translation of signs and actions  c. Home Tab has all the saved signs/actions which are previously translated

1. Sign Language Translation Models  

a. Static Signs 

i. Deep Learning Model For classification of different signs with no motion using MediaPipe + ML algo (to be  decided) / CNN / Yolo . 

ii. Integrating It to app through TensorFlow lite Framework and MediaPipe’s implementation or make an API using  Python and deploy it. 

b. Action Recognition 

i. Deep Learning Model for action (signs with motion) recognition using MediaPipe + ML algo (to be decided). ii. Integrating It to app through TensorFlow lite Framework or make an API using Python and deploy it. 2. Text Processing Model 

a. Raw text generated from translation models are converted into meaningful sentence. 

b. Most probably we will use Recurrent Neural Network for text processing. 

3. Mobile Application  

a. Login/ Sign Up for each individual using the app to keep track of all translations. 

b. Camera Tab Provide real-time translation of signs and actions 

c. Home Tab has all the saved signs/actions which are previously translated

Final Deliverable of the Project Software SystemCore Industry ITOther Industries Education Core Technology Artificial Intelligence(AI)Other Technologies Big DataSustainable Development Goals Good Health and Well-Being for PeopleRequired Resources

The main objective of this project is to design a system that can fill up the gap between the normal person and the impaired people and assist them to communicate with normal people in a comparatively better fashion.  This project objective:  i. Developing a sign/gesture recognizing system that can recognize sign gestures of deaf and mute people of American Sign  Language (ASL)  ii. The sign language will then be converted into text with the functionality of translation and prediction of the correct meaningful sentence.   iii. To work on better accuracy and performance of the system.