Smart Visual Guidance System For Visually Challenged People

Summary Smart Visually Guidance System for Visually Challenged People is a smart device that caters to blind or visually impaired people. This device will be made user friendly which will only require the person using it to press a single button multiple times

Project Title

Smart Visual Guidance System For Visually Challenged People

Project Area of Specialization

Artificial Intelligence

Project Summary

Summary

Smart Visually Guidance System for Visually Challenged People is a smart device
that caters to blind or visually impaired people.

This device will be made user friendly which will only require the person using it to press a single button
multiple times to select different options like multi Object Detection, its distance,
orientation, scene detection, and warning system. 

The information about the object in In front of the visually challenged people will be given to him using TTS
(text to speech) function through a camera which will allow any visually impaired person to live the same life as a non-visually impaired person.
Moreover, we will integrate the Google Assistant which will help the blind user to
recall his/her events that were observed from the headset earlier.

Currently, we are working on scene detection in two aspects.
1. Detection of all the exterior of NEDUET departments.
2. Detection of everyday objects with the help of COCO dataset.

To further take this FYP to the next step, we are also working on the following using a monocular camera only, without the aid of any other sensor:

1.Object Detection.
2.Object Distance Calculation.
3.Object Orientation Calculation.
4.Object Dimension Calculation.

All these features combined will allow any blind or visually impaired person to be able to sense their environment with great details and live a normal life just like other people with great ease and comfort in their lives.

Project Objectives

Specific Objectives Being Addressed by the Project:

1. Develop and research methods for identification and classifications of places using deep learning.

2.Research and study about deep learning tools like CNN, RNN, Tensor flow, and others.

3.Gather and manage a dataset for different categories activities, locations, to develop the required AI trained model using that dataset.

4.Study, research and create algorithms relating to object detection, classification, distance detection, object orientation, scene detection, and dimension detection.

5.Create and deploy Chabot services.

6.Integrate cloud services.

7.Google voice assistant integration.

8. Integrate all of our features into a single program and deploy it on the hardware for deep learning and AI model training.

9.To create a user-friendly product that can be easily operated by a blind or visually impaired person through just a single button.

Project Implementation Method

Project Implementation

1.Data Collection Strategy

Our target is to collect the dataset, for which we have to train a baseline model. The dataset consists of the images of all the exterior departments of NED.

We have an objective to collect 8000 images of all the exteriors of departments out of which 2700 images are of noon, 2700 of the evening, and the remaining 2600 are of dusk.

The images of each department will be taken from the front, right, and left sides. Moreover, the images from these orientations will be taken from 3 relative distances i.e near, near-far, and far.

2.Algorithm details and working

We are using the object detection model from Kaggle for object classification.

We are applying the perpendicular distance formula method and triangular approach to find out the distance of that object.

Further, we will be using the midpoint formula across the bounding box of the detected object to calculate its orientation and dimension. 

Finally,the waythe Scene Captioning works is known as “Image Caption”, a method to form sentences from
images. This technology is driven from a paper called “Show and Tell: A Neural Image Caption
Generator” which states that a ‘’generative model based on a deep recurrent architecture that
combines recent advances in computer vision and machine translation and that can be used to
generate natural sentences describing an image. 

3.Chabot Deployment Strategy:

We will develop a chat-bot using Google service “Dialogflow”. This chat-bot will be intelligent enough to understand human natural language and respond to human queries.

The chat-bot can be prompted to respond by asking questions like “Hey bot! Can you please recall what I had seen yesterday”.

4.Cloud Services Implementation Strategy

We will use Heroku/Digital Ocean for hosting our bot.

Further, we will use MongoDB as a database for storing all the memories observed by the device.

5.Google Voice Assistant Implementation Strategy

We will integrate that chat-bot with Google voice assistant on Android phones so that it can be easily accessible to the blind as android mobiles are cheaper and also contain Google voice assistant

6.Synergy Of All The Above Features

In the final approach, we will be deploying our model on an edge device i.e. Raspberry pi 4B+ . The camera will take input connected to the Pi. Further, that image will be preprocessed for Insights and produce the output in raw text, which will be then converted into auditory feedback using TTS (text to speech).

In the meanwhile, the resulting output will be transferred to the cloud database integrated with Google Assistant.

Benefits of the Project

Benefits of the project

With the help of this project the subject blind will be able to:

1. Recognize a scene in an image, for example, a girl playing Frisbee or a boy playing in the garden. In the early era, this seemed to be an impossible task for a computer to compute but due to the advancement in computer vision and deep learning, we can easily achieve this.

2. Identify distant objects as it seems to be a difficult task for a person to proceed towards something that cannot be felt with a stick.

By calculating the distance and other parameters like orientation with respect to the person and alerting him/her whenever a certain threshold of distance is triggered. This is to prevent further consequences. The system will look as shown in figure

3. Recall all his/her events that he/she that the headset worn by the user detected. This feature will help a blind person to recall past events from the diary.

4. Guide any blind or visually impaired person throughout the main branch of NEDUET.

Technical Details of Final Deliverable

Deliverables

Task1: Object detection, classification, and distance detection with a single camera (Duration: 2 months)

Collect the data for AI learning and model training for Object detection, classification, and distance detection.

Task2: Object orientation and dimensions detection with a single camera (Duration: 2months)

Data acquisition for the AI learning and model training for Object distance detection with a single camera.

Task3: Scene detection (Duration:3 month)

Acquire the data for AI learning and model training for Scene detection.

Task4: Chatbot development (Duration: 2 months)

Develop an interactive chatbot environment that will communicate with the person and provide any data upon request through voice.

Task5: Cloud service setup and Google Voice assistant integration (Duration: 2 months)

Establish a transfer and receiving mechanism for all the data generated by the trained model with the cloud service.

Task6: Final Hardware assembly and integration with software (Duration: 2 months)

Integration of all the software features and its deployment on the hardware along with real-time detection capabilities

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Health

Other Industries

Medical , Others

Core Technology

Artificial Intelligence(AI)

Other Technologies

Cloud Infrastructure, Wearables and Implantables, Others, Big Data

Sustainable Development Goals

Good Health and Well-Being for People

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com