Larynx - A Silent Speech Interface

Project Title

Larynx - A Silent Speech Interface

Project Area of Specialization Artificial IntelligenceProject Summary

The objective of the Final Year Project “LARYNX” is to develop a hardware-software system to help patients whose voice box has been removed to communicate by recognizing the signals produced by their brain. The hardware-software system will comprise of a specialized sensors and an android phone.

Due to consuming betel nut and tobacco over long periods of time, the throat is adversely affected which may result in throat (or laryngitis) cancer. This disease is quite prevalent in Pakistan. The treatment is laryngectomy i.e. surgical removal of the voice box. After the above treatment, the patients’ life is saved but they lose their ability to speak.

Our project LARYNX takes advantage of Sub-vocalization technique which may be described as silent articulation of speech, without moving the mouth or the jaw. When a person speaks, the brain sends neuromuscular signals to the facial muscles and the vocal cords to produce sound. Our project focuses on capturing signals produced in the patient’s brain using surface sensors, recognizing them using deep learning techniques and finally synthesizing an audio of the recognized word on an android application. Thus, the patients are able to have a primitive level communication with listeners.

The software developed will enable the patients to communicate in Urdu and other Pakistani languages thereby benefitting patients from rural areas. For the future, the project has potential of assisting patients with other speech disabilities such as apraxia or dysarthria using the sub-vocalization technique.

Project Objectives

To objective of the project is to develop a silent speech interface that allows patients having Laryngectomy surgery, to communicate using sub vocalization

In order to achieve the above objective, the major outcomes of the project are defined as follows:

• To capture Electromyography (EMG) signals using Surface Electromyography (EMG) sensors
• To develop a data set and train a deep learning model to recognize subvocalized words
• To develop an android mobile application to synthesize the recognized words or phrases as an audio

Project Implementation Method

The methodology of LARYNX comprises the following development work:

(a) Experimental Setup

The Data Acquisition System “26T Power Lab System” by AD Instruments, USA will be used in the project. Experiments to gather data will be conducted in a controlled environment. There will be 2 pairs of Surface EMG sensors that will be attached to Laryngeal and Digastric regions of the patient’s neck.  The sensors will produce raw analog EMG signals that will be fed into the system for processing. 

(b) Signal Processing

After data acquisition, signal processing will be conducted using MATLAB in order to:

• Remove noise from the signal
• Remove unwanted frequencies and features from the signal to make it smooth
• Compensate for Delay introduced in the above steps.              

(c) Feature Extraction

Features will be extracted from the signals to take out distinct variables to classify the signals better. A modified version of the Mel Frequency Cepstral Coefficients to adjust to the frequency of electromyography will be extracted from the signals. Coefficients from each signal will be used to create the feature vector.

(d) Signal Labelling and Data Set Generation

Signal labelling is part of the supervised learning technique of machine learning. The extracted features will be labelled with the corresponding word and saved to the data set. Likewise, as new signals are generated for the same word with each new patient, the data set will continue to grow.

(e) Deep Learning Model

Long Short Term Memory (LSTM) model will be used for the classification of the words. An LSTM has been chosen as the feature vector is based on time series data. Parameters of the model comprise hidden nodes, time steps, dropout value and learning rate.

(f) Android Application

The end user will use an android application to output the recognized word in both textual and audio formats. The application will be bilingual including the English and Urdu Language. The main features of the application include:

• Textual Output
• Audio Output
• Male/Female Voice
• Saving Common Words
• Making Sentences from Words
• Sharing Sentences

Benefits of the Project

In Pakistan, a significant population is suffering from diseases like Oral Cancer, Laryngitis and Laryngeal Cancer. These diseases are caused by Betel nut and tobacco abuse.

In laryngeal cancer, the voice box of the patient needs to be removed in a procedure known as laryngectomy. Interviews with speech therapists and family members of patients revealed that the greatest difficulty patients face is communicating with others. LARYNX will ease communication for patients by allowing them to speak in a primitive way and thus ease their discomfort to some extent. 

Moreover, the bilingual (English and Urdu) feature of Larynx will provide patients the ability to communicate in their local language, enhancing convenience of the patients and their care givers. The project will be cost effective as a result of local development.  

References

[1]"Speech language therapy in Pakistan", DAWN.COM, 2019. [Online]. Available: https://www.dawn.com/news/880784. [Accessed: 31- Oct- 2019]

[2]M. S. Khan, F. I. Banway, M. Hussain, M. H. Arshad, N. Nisar and S. R. Shah, “Comparison of knowledge, attitude and practices of betelnut users in two socio-economic areas of Karachi” in Journal of Pakistan Medical Association, Oct. 2013. [Online]. Available: https://jpma.org.pk/article-details/4927

Technical Details of Final Deliverable

The final deliverable will be an integrated hardware - software solution.

Hardware

26T Power Lab System by ADInstruments: An existing hardware system consisting of a Data Acquisition System and Electromyography (EMG) Sensors.

Software

Mobile Application: Android based mobile application, compatible with all versions of the Android Operating System above version 4.0 Ice Cream Sandwich.

Web Service: Laravel based Web Service to connect the android application to the desktop application, running on Laravel version 5.4.

Desktop Application: C++ based Desktop Application integrating the softwares MATLAB and Lab Chart which will be the actual silent speech recognition software that includes the trained model for word classification.

Final Deliverable of the Project HW/SW integrated systemCore Industry ITOther Industries Medical , Health Core Technology Artificial Intelligence(AI)Other TechnologiesSustainable Development Goals Good Health and Well-Being for PeopleRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	45360
Electrodes	Equipment	4	1000	4000
Electrodes	Equipment	26	510	13260
Preparation Gel	Equipment	1	5500	5500
3D Printing	Miscellaneous	1	6000	6000
MATLAB Student Suite Licence + Toolboxes	Equipment	1	12700	12700
Linux Shared Hosting	Equipment	1	3900	3900