Virtual Eye

Project Title

Virtual Eye

Project Area of Specialization Artificial IntelligenceProject Summary

Motivation

The current estimate suggests that there are 1.25 million blind individuals of all ages in Pakistan. They want to be a normal life like us but because of their disability, they face many problems in their daily life. They need assistance for doing their house works and finding the objects. They face difficulties in identifying the currency many peoples cheat them. They are facing problems in restaurants and shopping malls because they are unable to identify the price of the product and not able to read the menu cards of the restaurants.

The problems are visually impaired person are mostly facing are below.

1. Facing difficulty in visualizing the daily routine household items.
2. Unable to read even small and important text like price tags, hotel room no etc.
3. No idea about what he gives to others and receives in return.
4. Unable to access the near objects.
5. Depending on other individuals for the help.

Project Overview

Our project name is “Virtual Eye” based on the solution of these types of problems. By providing a mobile application that assists people with visual impairment in analyzing their surroundings, currency identification, reading menu cards. A single platform mobile application will be created (specifically for android) which will have the ability to recognize the household object and assist in voice, identify the Pakistani currency by pointing the camera on them, reading the menu card or tags by pointing the camera on them and provide voice assistance in Urdu language.

Project Scope

The app can be opened on a mobile phone through virtual assistance. The app has three different modes Object recognition, Currency identification, and Text reading which changes by voice command.

The scope of the project is that the mobile app is developed with the help of Image Processing, Artificial Intelligence and Machine Learning. The app is capable to assist using voice command to recognize household’s objects in the surrounding, do text read of menu cards and price tags to recognize the text. Object recognition contain most common objects like; House hold objects (fridge, washing machine, keys, bottle, mirror etc.), Common fruits (apple, banana, mango, etc.) Common food items (pizza, burger, cake, sandwich etc.) Common vegetables (potato, onion, carrot, etc.). And distance between object and the device is the approximation of foot steps required to reach to that object. The images like prize tags, billboards, menu cards are used for text from images option. Currency identification will be applicable on Pakistani currency only. Output or any feedback will be given to user through voice and in Urdu language.

Project Objectives

Project Objectives

The “Virtual Eye” project will meet the following objectives:

Objective # 1: The user is able to recognize the household object and assist in voice.

Objective # 2: User will be able to quickly identify correct value of currency notes by pointing camera on that note.

Objective # 3: User will able understand correctly, what is written in the hotel menu card or what are the prices written on the label of different things and, other short text by using our application.

Objective # 4: Enhance object detection feature so that it will now give the approximate distance between the object and the device.

Objective # 5: Solving the occlusion problem in Object recognition.

Objective # 6: All things will be done offline without any internet decency so that user can use the application anywhere and anytime.

Objective # 7: User will get quick response against each action in clear and loud voice.

Objective # 8: Fast and clear image captured by the camera will go under processing to recognize objects in it and user will get object name within 5 seconds.

Objective # 9: User can know how far his/her things are from him in footsteps.

Objective # 10: User will be able get pictures of objects and their recognition fast and quickly.

Objective # 11: Blind/visually impaired persons live will become 60% to 70% independent.

Project Implementation Method

In the implementation of the object recognition feature, we use Tensor Flow lite instead of Tensor Flow in this application because mobile doesn't support the Tensor flow API. We will do the training of the model using Python Programming Language. The name of the model is Mobilenet_V1_1.0_224 for Object recognition. And we will develop our Android application using XML and Java in Android Studio. For better performance and accuracy we are preparing our own dataset which we also customize according to our choice. In currency recognition mode the user pointing the camera to the currency note the system assists them through voice the note is 10 rupees, 20 rupees or 1000 rupees. For currency recognition, we will use the image classification technique to recognize the currency. Since we are dealing with reading short text like labels, price tags, menu cards Tesseract OCR is doing well in detecting these text. However, commercial APIs are better but due to budget limitations we will use Tesseract OCR

The Visually Impaired people access the App by using the "android accessibility suite ". There is only one (1) screen in our project which has three (3) different modes. The user can navigate between the modes with the help of the “android accessibility suite” and buttons. There are three (3) round buttons on our screen. The first is for a transition to the “Object Recognition” mode. The second button is for “Currency Identification” mode. While the last button is for “Text Reader” mode. The default mode is the first mode which is Object Recognition.

1. In the implementation of the object recognition feature, we use Tensor Flow lite instead of Tensor Flow in this application because mobile doesn't support the Tensor flow API. We will do the training of the model using Python Programming Language. The name of the model is Mobilenet_V1_1.0_224 for Object Identification. And we will develop our Android application using XML and Java in Android Studio. For better performance and accuracy we are preparing our own dataset which we also customize according to our choice.

1. In currency identification mode the user pointing the camera to the currency note the system assists them through voice the note is 10 rupees or 20 or 1000 rupees. For currency recognition, we will use the image classification technique to recognize the currency.

1. In Text Reader mode for reading short text like labels, price tags, menu cards Tesseract OCR is doing well in detecting these text. However, commercial APIs are better but due to budget limitations we will use Tesseract OCR

Benefits of the Project

The aim of this project is to make lives of visually impaired people easy by making an android app that helps them in doing daily life task. There is a need of this type of solution as many visually impaired people are facing problems in their daily life tasks. They always need someone’s help in crossing streets, finding things in home or anywhere, reading prize tags menu cards etc. The blind persons are the part of this world and we should treat them as a normal person but, they need someone in home to help them in their daily works and to pick something, with the help of this application they are able to do lots of work without the help of any person like recognition household objects, reading menu card, price tags and currency recognition. This application helps to overcome most of their daily life challenges which make them independent and more empower.

The project goal is to make blind peoples like independent by providing remote and automate assistance in three areas: object recognition, small text reader and currency identification. They will also give the following benefits:

• User will be able to quickly identify correct value of currency notes by means of a picture of that note.

• User can know how far his/her things are from him in footsteps

• User will able understand correctly, what is written in the hotel menu card or what are the prices written on the label of different things and, other short text by using our application.

• User will be able get pictures of objects and their identification fast and quickly.

• User will never have to learn foot counts or sights in home to put things their he/she can now know how far which thing is place in home.

• No need of other’s assistance to order food they like by reading menu by themselves or buy things of their favorite brands and follow warnings and precautions rule that are labeled by hearing the text of them with our text reader.

• They will never hesitate to do transaction in cash because of quick and reliable identification of currency.

• User will get quick response against each action in clear and loud voice.

• All these will be done at any place and anytime without any external support.

• Blind/visually impaired persons live will become 60% to 70% independent.

Technical Details of Final Deliverable

Technology Domain: Mobile Application                                 Development Language: XML, Java, Python
Model: SSD Mobilenet_v1

We are using the above mentioned tools and technologies to create our project. API’s are very helpful to perform the training and testing also the storage as well. We have to use minimum space for saving the large amount of data for the accuracy that is why we are using the technologies which play a tremendous role to complete that task.

There is only one (1) screen in our project which has three (3) different modes. The user can navigate between the modes with the help of the “Android accessibility suite”. There are three (3) round buttons on our screen. The first is for a transition to the “Object Identification” mode. The second button is for “Currency Recognition” mode. While the last button is for “Text Reader” mode. The default mode is the first mode which is Object Identification.

The application will have 3 modes for object recognition, currency identification, and image reading. The first mode is, recognize objects irrespective of their orientation and to some extent their size. And also, multiple objects in single image can also be detected. Dataset have more household objects for better object recognition. The second mode is for currency identification which will use to identify currency. Based on the given picture the app finds the exact match on the basis of color, number and other key points and, after finding the highest match voice-based feedback is provided. The third mode is to read text from the images. Output or any feedback will be given to user through voice and in Urdu language.

Final Deliverable of the Project Software SystemCore Industry HealthOther IndustriesCore Technology Artificial Intelligence(AI)Other Technologies OthersSustainable Development Goals Good Health and Well-Being for PeopleRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	11200
Google Play Developer Account	Equipment	1	4000	4000
FYP Report Printing	Miscellaneous	6	1200	7200