Language Learning Application for Children with Special Needs

Project Title

Language Learning Application for Children with Special Needs

Project Area of Specialization Artificial IntelligenceProject Summary

Our project is a Language Learning Application catered towards children suffering with Autism Spectrum Disorder (ASD). ASD renders these children semi-verbal to non-verbal. The focus of our application is to help these children to communicate effectively in everyday situations, and also help them learn basic to complex language skills. In developed countries, Augmentative and Alternative Communication (AAC) devices are used to cater to semi to non-verbal individuals. Unfortunately in developing countries like Pakistan, such softwares are rarely used due to high cost and language barriers.

Our application will be the first of a kind low cost AAC device which will incorporate the use of our in-house built Artificial Intelligence Language Models to cater to the child's needs. Furthermore, our monthly reports will provide value for the therapists, who can use of reports to monitor whether the child is progressing in his/her language skills or not.

We plan to distribute our application to the public, so that anyone whose child is suffering from such intellectual disability can use it. We also aim to incorporate our application into schools which cater to these special children, as this will allow for early adoption of language skills and hence enable effective learning from an early age.

Project Objectives

Our application aims to create a social impact on the lives of children suffering from Autism Spectrum Disorder (ASD), dyslexia, cerebral palsy, and other intellectual disabilities, which weakens their spoken comprehension skills. The solution to dealing with these children is through Augmentative and Alternative Communication (AAC) devices. Our objectives are:

• Low Cost AAC device. currently , our competitor analysis shows us that existing AAC devices are standalone handheld devices, average more than $500 each, while softwares start from $250.We aim to provide a smartphone/tablet application which will free of charge, as we know Pakistani market can’t afford to pay high amounts.

• Communicate effectively through our AI Language Models. We will use our own built AI Natural Language Generation model to convert random selection of word(s) into semantically and syntactically correct sentences. A user (child) can select any combination of words and our AI engine will generate logical sentences regardless of the selection sequence. This encourages the user to randomly select word icons representing ideas or comments that can be converted into sentences for self-learning and communication.

• Unintentional Learning. The primary goal of the child using this application will be to communicate effectively. But when sentences are generated, the Parts Of Speech (P.O.S.) tags are also identified to the child, allowing the child to learn the structure of a sentence. The sentence is also converted to audio, as research shows we learn best when words are associated with sound. So while the primary goal of the child is to communicate, he will also unintentionally learn language skills as well.

• Interactive Icons and Interface. Our application increases iconicity (how easily and completely a symbol icon represents a word for user understanding) by animating difficult words (verbs, emotions, and concepts like follow, lonely, and their) for easy understanding by intellectual disabled children.

• Minimal Literacy Application. Currently existing such devices and applications require a basic understanding and of English language, and a certain amount of literacy to use. This restricts the user involvement. Our application will be interactive, and will require minimum literacy to use.

• Monthly Progress Reports. We will generate a monthly report for the parent which will contain analysis of the words and icons chosen by the child, and the respective domains (situations) they belong to. Furthermore, we will also prepare a quiz from the generated sentences for the child. A therapist can use this to monitor the child’s evaluation.

Project Implementation Method

Our project consists of a number of different modules, and we’ll describe the implementation method for each below:

• AI Language Models: We wanted our language models to be able to generate logically correct sentences based on the words (constraints) given. This falls under the research domain of “Constraint Based Generation”. We implemented these Machine Learning models using Keras API, running Tensorflow in its backend. This allowed for fast experimentation and hence we were able to obtain high quality models.

• Cloud Deployment: We wanted to host our ML models someplace where it can be accessed by anyone around the world. So we decided to deploy it on Amazon Web Services (AWS) cloud. We made a Flask application in python which hosted our generator code. This flask application was hosted on AWS Lambda Function, which is serverless. This lambda function was triggered using AWS API Gateway. Our ML models were hosted on Amazon Sagemaker, and finally our word vectors were hosted on AWS DynamoDB. This architecture allowed for easy and quick inference whenever the user sent a query.

• Natural Language Understanding (NLU) component: Our NLU component had the essential function of identifying the parts of speech of each sentence, and then displaying that to the user as well. This was done using NLTK library, and was deployed on AWS using another flask application.

• React Native Application: We decided to build our smartphone/tablet application using React Native framework, running on top of Javascript. This gave us a number of essential benefits, but mainly it allowed us to make one application, and then use the same code to make Android application and IOS application. For the backend, we used PHP. MySQL database was used to store information about each user and his/her each sessions, so that monthly reports could be generated.

• Report Generation Module: At the end of each month our Report Module collects data from the user from the MySQL database, and then performs data analysis techniques on the obtained data, and then generates a customized PDF report for each individual. This is then emailed to the parent and/or the child’s therapist.

Benefits of the Project

Currently in Pakistan, there are roughly around 1.4 million children who might be suffering from some intellectual or neural disorder, resulting into potential disability. According to conservative estimates, more than 500,000 children are non-verbal in Pakistan. Most of these are lifelong disabilities without a cure. But with timely help, these children can adjust well into our functioning society.

Our Language Learning Application will help these semi-verbal to non-verbal children better communicate in everyday situations. They will be able to express their thoughts and ideas effectively. Not being able to say what you want leads to anxiety and frustration in the child. The child will also not be socially cast out, and will be able to blend in with other people. Our application gives the children an opportunity for empowerment and social inclusion through formal education.

Better communication leads to lower hit-and-miss scenarios, where the parent or caretaker has to guess as to what the child is trying to say.

Furthermore not being able to communicate with your teacher means you can’t ask questions and clear your doubts. This leads to ineffective learning in early school years, which are essential foundations of further education. Our application can integrate easy within schools and allow the teacher and child to communicate with ease and lead to better learning.

The reports that we generate contain important information about the child’s interaction with our application, and the reports and quiz that we generate provide great value to the child’s therapist, who can properly monitor the progress and growth of the child’s language skills.

Technical Details of Final Deliverable

Our Machine Learning models implied a one-to-many architecture in text sequences. Before our work, such architecture was mostly prominent in images, more specifically in “caption generation” tasks. Textual data incorporated a seq2seq, in which the input given is a sequence, and the output is also a sequence. We wanted an architecture which could give us as output a sequence, but as input it would only take in one word, and not a whole sequence.

So our first approach was using N gram models to cater to this issue. And then we tried Hidden Markov Models (HMM’s). But we soon realized that both of these methods suffered predominantly from the same issue, which was “variable context capture” issue.

We decided finally to use Recurrent Neural Networks (RNN’s) to cater to our problem, and use a variant of RNN’s called Long Short-Term Memory (LSTM) networks, which are the state-of-the-art when it comes to generating sequences with variable context.

Our novelty came in how we decided to basically tune our dataset. Since we wanted to generate sequences from one word, we had to make sure that our model was trained on that specific word. But at the same time, we also had to make sure that our model generated semantically and syntactically correct sequences, and in order to do this we also had to capture the positional information of the word. To cater to the position information, we decided to split our dataset and train it on verbs. Verbs are actions and these are mostly used in everyday scenarios to describe what you want to do, what you need, your emotions, etc. So we decided that verbs would be the perfect place to start. So we divided each sequence in the training dataset into two parts; 1) first part containing the sequence up till the verb, 2) second part containing the remainder of the sequence. Then we inverted the first half of the sequence to train our Backward Language Model. The second part of the sequence was sent to be trained on our Forward Language Model.

Then after sentence generation, if we had another constraint to incorporate apart from verb, we would use cosine similarity to between word embeddings to calculate the closest word in the generated sequence to replace the other constraint with.

Final Deliverable of the Project Software SystemType of Industry Education , IT , Health Technologies Artificial Intelligence(AI), Cloud InfrastructureSustainable Development Goals Good Health and Well-Being for People, Quality EducationRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	70375
HP 23.8	Equipment	1	21000	21000
ADATA DDR4 RAM	Equipment	3	6500	19500
WACOM Intuos Art Tablet	Equipment	1	22100	22100
Visit To Psychologists, Therapists and Autistic Children.	Miscellaneous	5	500	2500
Design Expo Photo-paper Printouts	Miscellaneous	20	60	1200
Design Expo Flow Diagram Prints	Miscellaneous	20	15	300
Final Defense Binding Charges	Miscellaneous	2	500	1000
Final Defense CD-R Charges	Miscellaneous	3	100	300
Final Defense Colour Printing	Miscellaneous	23	25	575
Final Defense Normal Printing	Miscellaneous	80	5	400
Project Poster Print in 2x3 Feet	Miscellaneous	1	1500	1500