FPGA Based Lane Detection And Tracking System For Autonomous Vehicles

Project Title

FPGA Based Lane Detection And Tracking System For Autonomous Vehicles

Project Area of Specialization Artificial IntelligenceProject Summary

There have been great advancements in the field of computer vision due to the utilization of machine learning. Nowadays the role of image processing is demanded in a wide range of applications. Image processing units are embedded in many devices ranging from mobile phones to Autonomous cars. However, the processing speed of some complex algorithms puts a barrier in a real-world implementation. Applications like, autonomous driving or industrial automation are subject to tight real-time constrain and low latency requirements. For these applications energy efficient and faster speed platform is needed. Nowadays, GPUs are used as brain for image processing. However, in comparison with GPUs, FPGAs can deliver superior performance in deep learning applications. The main advantages of FPGAs are faster image processing over GPUs, CPUs and re-programmable capability over ASICs. However, designing machine learning applications and their deployment on FPGAs is very challenging because FPGAs are hardware programmable and therefore require hardware expertise to design such type of applications and is also time consuming. Idea of this project is to design an algorithm for lane detection and tracking system for autonomous car using high level programming and deploy it to FPGA without direct hardware designing and need of hardware expertise. This is achieved by designing DNN using ZyNet package of python and get HDL code from there. Then synthesize the HDL code, simulate it, implement and generate bitstream file and upload on Zybo Zynq-7000 SoC. Finally compare software implementation results and hardware implementation analyze the results for both.

Project Objectives

• Data acquisition (dataset). 
• Pre-processing & Dataset loading.
• Generation of weights and biases.
• Development of DNN for lane detection and tracking system using ZyNet package of Python.
• From the model trained (using python), generate synthesizable HDL code.
• Simulation, Synthesize and Implementation of model on Zybo Zynq-7000 SoC using Vivado software.
• Testing & analysis

Project Implementation Method

Initially, we need to acquire the data i.e., we collected the dataset. Initially we took CuLane dataset which is a large-scale challenging dataset for academic research on traffic lane detection. We customized the dataset since it is very large dataset. After that the data is preprocessed and loaded. After successfully loading our dataset, we need to generate weights and biases that are required to train our model on ZyNet (a Python package), in order to get a synthesizable HDL code. We can generate weights and biases by training our model on TensorFlow or we can generate weights and biases by another way in which we need to build the neural network using high-level programming language particularly Python, train our neural network and save the weights and biases. By generating weights and biases by any of the above mentioned methods, we also get software implementation results that we later on will compare with those of the hardware implementation. Once we get weights and biases, we train our neural network using ZyNet package of python in which we will use the generated weights and biases. We use ZyNet, a python package, to generate a synthesizable HDL code (Specifically Verilog code) for Zybo Zynq-7000 SoC. Other Xilinx FPGA devices are also supported such as Zedboard etc. ZyNet cannot only make Xilinx Vivado project files but it can also package the entire neural network in a single IP and it can also make system by interfacing our neural network with PS (processing system). Once we get HDL code then using Vivado software we synthesize our HDL code and write the testbench file to simulate the project. After simulating the HDL code, we implement it by translating, mapping and placing and routing it. After that we can get the hardware implementation results that we will compare with those of software implementation that we get from training our model on GPU.

Benefits of the Project

Benefit of this project is that it allows machine learning developers to develop an algorithm using high-level programming language Python and deploy it on FPGA. FPGAs are flexible and handle complex algorithms better in real-time environment that makes them a good choice for Autonomous vehicle/ADAS (Advance driving assistance system). Main advantage of FPGAs is faster image processing over GPUS, CPUs and reprogrammable capability over ASICs. Automotive companies spend large amounts of money to test Autonomous cars and by using these reprogrammable devices they can test their designs at faster speeds and the time to market can be reduced.

Technical Details of Final Deliverable

The final deliverable will have simulation results that will provide hardware implementation results that we will compare with those of the software implementation in order to highlight the significance of FPGAs. We also plan to design a prototype by interfacing Pcam 5C Camera module and monitor display with the Zybo Zynq-7000 SoC and fit the design on a robotic/miniature car to test real-time lane detection and tracking system on FPGAs. Therefore the final deliverable will have a prototype including miniature car, Pcam 5C camera module, Zybo Zynq-7000 SoC and monitor display.

Final Deliverable of the Project HW/SW integrated systemCore Industry TransportationOther Industries Others Core Technology Artificial Intelligence(AI)Other TechnologiesSustainable Development Goals Decent Work and Economic Growth, Responsible Consumption and ProductionRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	64858
Zybo Zynq-7000 SoC	Equipment	1	46327	46327
Pcam 5C Camera module	Equipment	1	8391	8391
Pmod OLEDrgb: 96 x 64 RGB OLED Display(Monitor display)	Equipment	1	3740	3740
Thesis Printing	Miscellaneous	4	1600	6400