An Autonomous Shopping System Using Vision And Sensor Fusion Techniques

Project Title

An Autonomous Shopping System Using Vision And Sensor Fusion Techniques

Project Area of Specialization Artificial IntelligenceProject Summary

INTRODUCTION:

The Project Title is "An Autonomous Shopping System Using Computer Vision and Sensor Fusion Techniques". The scope of this project is to make shopping groceries usable and faster without waiting in line for checkout. The emerging technology-based devices will track their purchases, allow them to exit the store without a physical checkout, and automatically charge their account. This type of technology uses computer vision, deep learning algorithms, and sensor fusions. When a person enters the store, the cameras imbued with computer vision will identify the person and start tracking his/her movement. The shelf Proximity sensor will send the data for a person who is standing beside that shelf. As the person picks an item, the pressure releases from the Proximity sensor and the item  will be added to his/her cart, automatically to the buffer.If the person changes his/her mind and put back the item to the corresponding shelf, the Proximity sensor reverts back to the previous value and the item will be discarded from the bill. If the person checkouts from the shop as detected by Proximity sensor, the total amount is calculated and the bill is displayed .

PREVIOUS WORK ON THIS PROJECT:

• A number of methods has been proposed by researchers in this domain. High Speed Billing System is proposed in which the RF detector is placed inside the shopping cart which is linked to the server for billing
• Another project proposed Smart Trolley in which they used RFID cards for each product and RFID reader with MCU on each trolley for calculating the bills while shopping.
•  The Smart Shopping Cart was proposed with Zigbee and RFID in which they utilized RFID cards for each product along with Product Identification Device (PID) for the trolley which is used for calculation of products and bill. This approach used Zigbee for transmitting the billing details to central billing system .

DIFFERENCE B/W PREVIOUS WORK AND THIS PROJECT:

This project is completely different from the previous works as we are neither scanning any product manually nor any cart system with RFID is employed. We have proposed a method that works with webcam imbued with computer vision techniques, location sensor using Proximity  and pressure sensors.  The sensor fusion techniques are used for the automatic billing, therefore,  no need of any manual work is needed.

PROBLEM IDENTIFICATION And NEED:

• We are trying to solve the problem of long checkout lines in shopping malls which is a wastage of precious time as well as energy. We want to make shopping more comfortable and task focused. We want to introduce a future shopping user friendly environment.
• The need of this technology is to adopt thest  technological advancement in such a way that not only it can be accepted by the consumers but also is applicable to Pakistani shopping environment. Furthermore, new paradigms can be explored while working on this project in terms of technology, new trends, and entrepreneurship.

Project Objectives

Aim of Project:

To design a comfortable and digital automatic shopping environment for a user to omit the time for billings or checkouts.

Objectives of Project:

• To survey the existing works for automatic billing in the shopping malls or convenience stores.
• To design an empirical framework based on the sensor fusion for automatic billing.
• To strive for the realization of the proposed empirical framework with a prototype.
• To evaluate the system’s performance for single as well as multiple users.
• To evaluate more complex tracking algorithms for validating the feasibility of the system.

Project Implementation Method

• It starts with face detection of person and that signals goes to Raspberry pie microcontroller that allow person to enter in the shopping mall.
• camera starts tracking person each and every moment.
• if person  comes to the Rack area the proximity sensor of that  shelf  will have some respond and if 1's in response of any particular sensor check the shelf Id of that sensor we get person location in Rack area.
• After that Person take any product the proximity sensor of that product released some pressure then add the particular products to buffer (temporary).
• Now person wants to leave the store then check the status of pressure if still we have pressure released then check Barcode of that product by Barcode reader that is embedded in Rack and if person change its mind and pressed the product then we have pressure status released to original value then remove the items to the buffer.
• Then calculate the total bill of the products and display the bill has be deducted.

Benefits of the Project

Time Efficiency:

• The biggest advantage is time efficiency.
• The whole idea behind this project is that you grab what you need, and leave.

Smart technology:

• This project will be a steppingstone for the consumers to experience the current trends and technologies and interact with the environments available in the developed countries.

Artificial Intelligence

• Currently, as we are aware that our government is focusing on the use and learning of Artificial Intelligence, extensively. We are introducing the use of AI in the local environments with everyday consumers.
• This will provide a kind of feedback if such technology is ready to be implemented and accepted by the masses of this country.
• It will also highlight the concerns associated with the use of advanced technologies such as security threats, privacy constraints, and sensor fusion problems.

  Social Benefits

• Help relieve stress of shopping
  • One of the important benefit is reducing stress level of a customer as this system will help avoid long queues and making the shopping environment more comfortable and friendly.
• Once, system like this gets publicly accepted and integrated in the major shopping malls, following benefits can be noticed.
  • Ease of use
  • New Job Openings (in terms of security specialists and HR personnel for setting ethical guidelines)
  • Taking step ahead by regulating AI in Pakistan.
  • Making the public aware with the positive use of current AI technologies, i.e. changing the perspective.
  • Increasing the reach and diversity of the products, i.e. products from small companies can get the same amount of attention.

• Future scope:

• In future, this technology can be implemented in multiple business sectors such as farming, industries, security agencies, and so forth.
• In future, this technology can be employed at railway stations and bus stands for automatic ticketing.  

Technical Details of Final Deliverable

Technical details:

• We start with face detection and we intend to use the state-of-the-art FaceNet technique for user identification.
• After the face is detected the identification is performed at Raspberry where the trained model for facial recognition is saved in order to perform the real-time inference.
• Once recognized, the cameras will track the person continuously.
• The sensors Fusion streaming data goes to ESP8266 WIFI microchip and this chip is connected to Raspberry Pi microcontroller .The ESP8266 WiFi Module is a self contained SOC with integrated TCP/IP protocol stack that can give any microcontroller access to your WiFi network. It offers a complete and self-contained Wi-Fi networking solution, allowing it to either host the application or to offload all Wi-Fi networking functions from another application processor.
• customer indoor tracking purpose we are using beacon sensor that are always on-transmitter used BLE(bluetooth Low Energy) that broadcast signals to near by portal devices like mobile and tablets.
• Sensor shelf are equipped with the proximity and IR sensors to sense the presence of users as well as to check the placement of the products.
• A person picks up any item from the rack, the pressure sensor attached to each and every product of Rack so we have pressure released that’s mean somethings pick up and then Barcode Scanner of that shelf scan the product key of the item that is taken by person and if the pressure is still  released then add that items to the buffer.
• If the person changed its mind and return the items to Rack  so pressure return to its original position and then the barcode scanner scan item and remove  the items from buffer
• If person intentionally or unintentionally placed somethings else to the Rack instead of that pick up product the camera is continously tracking and barcode scanner detect the product barcode in case of mis match it acknowledge the system.
• If person picks up desired items from store and satisfied with shopping and want to leave the store the Infrared Radiation sensors that is implanted to the exit of store verify person location leaving the store so the items that is added to their account will be calculated and total bill receipt send to the person cellphone through GSM module and  then person can check its receipt at that time all desired products are present in receipt and by verifying okay text the person easily leave the store.

Final Deliverable of the Project HW/SW integrated systemCore Industry ITOther IndustriesCore Technology Artificial Intelligence(AI)Other TechnologiesSustainable Development Goals Decent Work and Economic Growth, Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	79960
Nest web Camera	Equipment	1	35000	35000
IR sensor	Equipment	21	120	2520
BMP280 Barometric Pressure Sensor	Equipment	15	300	4500
Raspberry pi 4	Equipment	1	8000	8000
Beacon Sensor	Equipment	1	2000	2000
Wifi Module-ESP8266	Equipment	2	350	700
Sim900a-GSM Module	Equipment	1	3000	3000
Rack For product Display	Miscellaneous	1	2000	2000
Poster for FYP	Miscellaneous	1	3000	3000
Thesis for FYP Project	Miscellaneous	1	5000	5000
Battery	Equipment	2	6500	13000
Arduino Mega	Equipment	1	1240	1240