Sign O Talk

Project Title

Sign O Talk

Project Area of Specialization Augmented and Virtual RealityProject Summary

Hearing and Speech Impairment is one of the most common disabilities in Pakistan and around 7.90% of overall population is affected with this defficiency.The major shortcomings of society is the social communication barrier between disabled and normal person. It’s hard to understand the sign language and fulfill the requirement of the disable person specifically in the commercial environment like client services. To overcome this communication barrier, the proposed application “Sign O Talk” will be used as a mediator to provide a platform in a conversation between a hearing and speech impaired person and a normal person the ease of communication and emphasizes an effective, efficient and portable application that can overcome the communication problem. The aim of the application is to convert the sign language into speech and text by using the Smartphone camera in real time and convert the text and speech into Sign Language. By comparison of existing applications “Sign O Talk” will be the first application that will decode the sign language into  Urdu Language words so this application can easily use by any people who couldn’t able to use English mediator. The “Sign O Talk” will focus to display the converted text and speech into Signs with three-dimensional animated hand movements avatar created by Augmented reality so an impaired person would be able to clearly see and understand the sign.

Project Objectives

The hearing impairment and inability to verbally express oneself leads to lack of equal opportunities as well as leads to problems even in everyday communication to solve the issue this application “Sign O Talk” has been proposed with the main objective to provide the easy and portable communication bridge. The main objective is to employ Natural Language Processing to communicate and decode the Sign language into Urdu Language to provide easy and understandable Sign language mediator. The Sign language user can able to understand the sign with the help of avatar. The normal person can use the camera of Smartphone to scan the sign which will decode into text and speech so he/she will easily understand impaired person. The impaired and normal people can easily communicate without any previous knowledge. To provide the portability to easily carry the application within your smart phones.

The hearing impairment and inability to verbally express oneself leads to lack of equal opportunities as well as leads to problems even in everyday communication to solve the issue this application “Sign O Talk” has been proposed with the main objective to provide the easy and portable communication bridge.

• The main objective is to employ Natural Language Processing to communicate and decode the Sign language into Urdu Language to provide easy and understandable Sign language mediator.
• The Sign language user can able to understand the sign with the help of avatar.
• The normal person can use the camera of Smartphone to scan the sign which will decode into text and speech so he/she will easily understand impaired person.
• The impaired and normal people can easily communicate without any previous knowledge.
• To provide the portability to easily carry the application within your smart phones.

Project Implementation Method

“Sign O Talk” will be developed by designing a two different interface one for hearing and speech impairment persons and second for normal person.

The interface of  hearing and speech impairment persons will display an avatar that will created by using AR which avatar will perform the sign decoded by speech and text that will understandable according to sign language. The avatar will be created by using unity.

The normal person interface will use Smartphone camera to scan the sign and display the meaning of the particular sign in text and speech. The Sign language will be decode by using the algorithm comparison with a specific database that will be design using SQLite which will store the information and meaning related to the particular sign in Urdu language.

The layout of the application will be developed by android studio

Benefits of the Project

The application is the complete portable solution to provide communication platform between hearing speech impaired person and normal persons The application will be designed to decode the sign language in Urdu language which is easily accessible. This application provides the sign into text and speech as well to provide a smooth communication even if the person is unable to read. This application will help impaired people to easily understand the text and speech converted to sign which are display by using avatar creating by Augmented Reality.

• The application is the complete portable solution to provide communication platform between hearing speech impaired person and normal persons
• The application will be designed to decode the sign language in Urdu language which is easily accessible.
• This application provides the sign into text and speech as well to provide a smooth communication even if the person is unable to read.
• This application will help impaired people to easily understand the text and speech converted to sign which are display by using avatar creating by Augmented Reality.

Technical Details of Final Deliverable

The Sign O Talk will be a mediator to convert sign into text and speech and vice versa in Urdu Language. Text and speech can be converted into signs and can be easily understandable by 3D avatar by hearing and speech impaired persons. The application will use the following technology; Digital image processing is the use of a digital computer to process digital images through an algorithm. In this application this technique will use via Smartphone camera to scan the image of the particular signs and decodes it into text and speech and vice versa. Computer vision is a technique that deals with how computers can be made to gain high-level understanding from digital images or videos. As image processing is a process that can cause delay to overcome this delay computer vision can be used. The haar algorithm of computer vision will be used which is for object detection to scan the sign and match it with the database to decode the words for providing communication. Augmented reality is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information. We are using the AR to decode the words into sign by creating an avatar so a sign language user would able to see sign in real time environment and decode the conversation. The application will use text to speech technique to make the application audible if someone cannot read so he must able to use this by the feature. This application will be developed by using android studio. It will be compatible with the mobile phone camera to provide portability as a mediator.

The Sign O Talk will be a mediator to convert sign into text and speech and vice versa in Urdu Language. Text and speech can be converted into signs and can be easily understandable by 3D avatar by hearing and speech impaired persons.

The application will use the following technology;

• Digital image processing is the use of a digital computer to process digital images through an algorithm. In this application this technique will use via Smartphone camera to scan the image of the particular signs and decodes it into text and speech and vice versa.
• Computer vision is a technique that deals with how computers can be made to gain high-level understanding from digital images or videos. As image processing is a process that can cause delay to overcome this delay computer vision can be used. The haar algorithm of computer vision will be used which is for object detection to scan the sign and match it with the database to decode the words for providing communication.
• Augmented reality is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information. We are using the AR to decode the words into sign by creating an avatar so a sign language user would able to see sign in real time environment and decode the conversation.
• The application will use text to speech technique to make the application audible if someone cannot read so he must able to use this by the feature.
• This application will be developed by using android studio. It will be compatible with the mobile phone camera to provide portability as a mediator.

Final Deliverable of the Project Software SystemCore Industry MedicalOther Industries IT Core Technology Augmented & Virtual RealityOther Technologies OthersSustainable Development Goals Good Health and Well-Being for PeopleRequired Resources

The application is the complete portable solution to provide communication platform between hearing speech impaired person and normal persons The application will be designed to decode the sign language in Urdu language which is easily accessible. This application provides the sign into text and speech as well to provide a smooth communication even if the person is unable to read. This application will help impaired people to easily understand the text and speech converted to sign which are display by using avatar creating by Augmented Reality.