Toll tax collection with image based vehicle type and license plate recognition system using deep learning

Project Title

Toll tax collection with image based vehicle type and license plate recognition system using deep learning

Project Area of Specialization Artificial IntelligenceProject Summary

Tolling efficiency in the manual toll tax collection system is very low and time-consuming. With the increase in traffic in recent years, a bottleneck situation is created on the toll plaza, which is the wastage of the time and fuel. This project proposes an automatic toll tax collection with image-based vehicle type and license plate recognition system (VT-LPR) in Pakistan's environment. The task of image-based vehicle type and license plate recognition becomes non-trivial due to image variations caused by several factors. These include cluttered background, partial occlusion caused by the dust and mud on the license plate, non-uniform lighting conditions, orientation changes depending upon the relative orientation of camera and vehicle, low resolution images, different font and font sizes of the characters in the license plate, and no standard license plates in Pakistan. Consequently, we propose to use state-of-the-art performing deep learning based algorithms such as YOLO (You Only Look Once) for Vehicle type recognition and License plate detection and extraction. The three staged pipeline consists of vehicle type recognition, license plate detection and character recognition. In order to train and test YOLO, we will gather an image dataset of 15k images of different vehicles (Truck, Car, Bus, Van) throughout Pakistan. The algorithm will be trained with the best results and will then be deployed on a Raspberry-Pi to work in real time. The image taken with real time deployment on the toll plaza will then be processed to recognize vehicle type and license plate and generates the toll tax amount. This information will be stored in the database file for record maintenance.

The proposed system will be utilized in traffic monitoring, security surveillance, access control and also for stolen vehicle identification.

Project Objectives

The following are the objectives of toll tax tax collection with vehicle type and license plate recognition system:

• Collection of large image data (15k Images) of different vehicles (Truck, Car, Bus, Hiece, Van) such that it depicts the license plate
• Data pre-processing (cropping, resizing)
• Image annotation for ground truth generation
• Train CNN famous object detection algorithm i.e. YOLO v3, for vehicle type recognition, license plate detection, license plate character segmentation and recognition
• Developing the Toll tax amount collection based on the vehicle type and license plate number
• Deploying the model on Raspberry-pi for real-time predictions and interfacing ultrasonic sensor for vehicle arrival detection.

Project Implementation Method

The following are the steps for the project imlementation:

1. Image data collection:

The data collection step is to collect images i-e. 15k images of different vehicles from different cities of Pakistan. Different type of vehicles are:

• Truck
• Bus
• Car
• Van

2. Data Pre-Processing:

Once image data of different vehicles are collected, these images are pre-processed by applying cropping, resizing to get a region of interest where the vehicle is present.

3. Manual annotation for ground-truth generation:

After Pre-Processing, we will annotate each image in 3 phases. Firstly, images are annotated for recognizing vehicle. Secondly, license plates are annotated for detecting LP. And Finally, each character is annotated for character segmentation and recognition.

4. Training and testing of object detection model:

Once the image dataset is prepared, our next step is to train object detection models for detecting objects like vehicles, license plates, and characters from license plate. The trained model is then test on the testing data for checking the performance of the trained model.

5. Vehicle arrival detection:

Vehicle arrival is detected through ultrasonic sensor. The sensor is interfaced with the Raspberry-pi. 

6. Real time prediction through Raspberry-pi plateform:

After the model is trained, we deploy the model on raspberry-pi for real-time predictions. Raspberry pi platform is developed for image and video acquisition. Toll amount is collected based on the vehicle type and their license plate number.

Block Diagram:

Flow Chart:

Benefits of the Project

Vehicle type and license plate recognition system can be utilized in several applications. This system can be deploy on the toll plaza for automatic calculation of the toll amount.  This system is further utilized in:

• Deploy on the Parking lots for the management of the parking security and  access information.
• Provide access control to the restricted or military classified areas. This will increase the security, logistic managements, event logging, database management.
• This system is an ideal technology for the surveillance purpose to make an intelligent traffic monitoring system.
• This system is very useful for the law enforcement agencies. They can use this system to identify the stolen vehicles.
• This system is also used to monitor the traffic rules violation.
• This system adds significant value in Border Control by maintaining the record of the border crossing, keep check on illegal activities.

Technical Details of Final Deliverable

-An automatic toll tax collection system.

             Camera resolution 1280x720

             Camera resolution 640x480

-Software or feature extraction, image enhancement, and vehicle detection

            Resized and cropped image.

             Feature extraction of vehicle detection, license plate detection and recognition using YOLO v3 object detection algorithm.

-Raspberry-pi based plateorm for realtime image and video acquisition.

              Linux based operating system.

              Used for Image processing, real time vehicle type and license plate detection and recognition. 

Final Deliverable of the Project HW/SW integrated systemCore Industry TransportationOther Industries Energy , Security Core Technology Artificial Intelligence(AI)Other Technologies Internet of Things (IoT)Sustainable Development Goals Industry, Innovation and Infrastructure, Sustainable Cities and CommunitiesRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	61630
Raspberry-pi 4B 4GB Ram	Equipment	1	17000	17000
Power module for Raspberry-pi	Equipment	1	2350	2350
Exhaust fan for Raspberry pi	Equipment	1	500	500
32 GB Micro SD card	Equipment	1	1600	1600
Self adhesive pure copper heatsink	Equipment	2	350	700
Long range ultrasonic sensor module	Equipment	2	2040	4080
HikVision CCTV Camera module	Equipment	1	25400	25400
Equipment required for assembling (wires, solder iron etc.)	Miscellaneous	5	2000	10000