Three dimensional vision autonomous quadcopter

Autonomous drones (Miniature aerial vehicles/MAV) in recent years have become an essential tool in the area such as aerial surveillance, visual inspection, military, remote farming, filming etc. Quadcopters are becoming popular all over the world  due to its small size, high manoeuvrability, st

Project Title

Three dimensional vision autonomous quadcopter

Project Area of Specialization

Artificial Intelligence

Project Summary

Autonomous drones (Miniature aerial vehicles/MAV) in recent years have become an essential tool in the area such as aerial surveillance, visual inspection, military, remote farming, filming etc. Quadcopters are becoming popular all over the world  due to its small size, high manoeuvrability, stability and agility enable it to fly in the low indoor environment. We will also use it to survey or reach such areas which are very difficult to access. For example, In Pakistan this drone can be used to supply medical drugs in northern areas immediately where remote access is difficult or impossible. Similarly, farmers in Pakistan will use this drone for their crop protection, care and maintence. 3-D vision and image processing application can be very useful in security as well as in military applications in Pakistan. This project will provide the foundation for all the above mentioned projects and bring revolution in drone technology in Pakistan.

In this project, we will develop a system that will enable the quadcopter to autonomously navigate the targeted location and avoiding the detected obstacles in its path. We will use the monocular camera and Lidar for the detection of obstacles, for localization and mapping as well as for navigation.. The autonomous quadcopter that we will design will be capable of self-controlled flight. We will use QC weight lifter which is capable of making a path using the Global Positioning System (GPS). Jetson nano GPU will be used to control the flight of our quadcopter and set the directions of our quadcopter. To implement that idea we will have three basics steps to follow. First, we will achieve the estimate of the quadcopter pose in the environment and also calibrate the control signal required to reach the targeted position in the map. To accomplish this localisation and mapping, we will implement the approach of autonomous camera-based navigation of a quadcopter. Second, we will use Oriented FAST and Rotated BRIEF (ORB) to extract features from the video frames and match the extracted features with the known features to classify if the object is an obstacle. Finally, given that the quadcopter pose, presence of obstacle will be known we implement the Function approximation and feature-based method of reinforcement learning to navigate to all the target locations defined avoiding the obstacle in the path.

We will implemente our approach on an AR. Drone and will test it in different environments. In our approach, all the computation of localisation and mapping, object recognition and reinforcement learning will be performed on a ground station. The algorithm of machine learning like Monocular SLAM, PTAM, Extended Kalman Filter and Reinforcement learning will be used to do localisation, navigation and path detection.

Experiments will have validated that the presented methods work in various environment and can navigate accurately to predefined target locations.   

Project Objectives

To navigate a quadcopter, estimation of the pose as well as the velocity, knowledge about the scale of the map and control signal that needs to be sent to the quadcopter to reach the target position are essential. For this purpose This project is divided into different significant components those are:

Localisation and Navigation of the Drone

Obstacle Recognition

Autonomous avoidance of obstacles

Developing an autonomous drone that can reach its desired location. This location is given to drone by setting the GPS location. The drone will reach its destination autonomously avoiding all obstacle in its path with the help of localisation and 3D vision. Thus named as 3D vision. This project is targeted to achieve the autonomous flight of drone in Pakistan that can be primarily be used to deliver food, medicine, first aid tool-kit and another necessary thing to the remote area where access by road is difficult. Pakistan northern area can use this drone in future to supply medicine quickly where needed, and these drones can also help the growing online food companies to expand and improve their business by using drones for delivery. This drone can provide to foundation base for future autonomous drone projects that can be used in the agriculture sector, in the railway sector and power sector.

Project Implementation Method

To achieve the task of localisation of the drone and enable the drone to navigate. This methodology is based on three major components which are listed below:

• Monocular SLAM:: For solving the SLAM problem, PTAM algorithm is applied to all the video frames, which enables it to estimate the drone’s pose. The scale of the map is essential to navigate the quadcopter, define the flight plan, calculating control commands, to fuse visual pose available from the PTAM with the sensor values, like ultrasound sensor values.

• Extended Kalman Filter:: For integrating all the sensor data provided from the drone, the pose estimation provided by the PTAM and the effect of the control commands sent to the drone extended Kalman filter (EKF) enhance the estimate of the drone’s pose and also gives a right prediction of the future state when the command is sent PID.

• PID controller:: For navigation of the drone to a given target position, the estimated velocity of the drone and estimate of the drone’s pose that is provided by EKF is used to calculate the required control commands using the PID controller.

Process of PTAM can estimate the pose, velocity of the quadcopter in the environment. After the pose estimation is carried out, the methodology provides the functionality of navigating the quadcopter to the target location. To navigate through the predefined target locations, the PID controller is provided with the target location then the controller calculates the required control signal that has to be sent to the quadcopter to reach the target location with the help of the pose estimation provided by the EKF.

To achieve the task of Obstacle Recognition to classify an object, first, it is required to train the recognition algorithm with known objects. The features extracted from these known objects are later used to classify if the object is an obstacle or not. After the obstacle is detected, it sends a notification to the reinforcement learning algorithm, which then sends the necessary action to avoid the obstacle. The quadcopter will have proper planning algorithm to navigate to the goal position and have knowledge of the obstacles for preventing a collision. Finally, avoiding the obstacle. LIDAR, along with radar sensor, comes handy here as it helps the drone to know the distance between from the obstacle and carefully avoid it. An algorithm made for obstacle avoidance purpose with four main feature those are target position, obstacle in the path, target position reached and distance to the target position.

Benefits of the Project

There are numerous benefits of autonomous drones in the industry as well as for milltary purposes.

It can be used to deliver various things within the city without requiring any human controlling. It can bring revolution in the food industry for providing food quickly. It can also deliver blood and medicines in case of an emergency in remote areas.

This drone can provide to foundation base for future autonomous drone projects that can be used in the agriculture sector, in the railway sector and power sector as discussed below.

The flight of this drone is entirely autonomous; it can help farmers to see the yield of crops and for spraying purposes which can reduce the cost merely to half.

For surveillance purpose, It can help be used for humanitarian and disaster relief missions. To help monitor and combat forest fires, surveillance drones outfitted with thermal imaging cameras are being deployed to detect abnormal forest temperatures. By doing so, teams can identify areas most prone to forest fires or identify fires just minutes after they begin.

Significant advantage at hand for developing this 3D vision project is that it can be used without the need of any GPS, localisation and avoiding obstacle using its 3d vision. It must be noted that such drones can be of an excellent asset for our Army. In the military, it can inspect borders to see if any vulnerability in a line of control more fastly and efficiently then human. Drones can also be mounted with weapons for fighting purposes. A swarm of drones with weapons loaded is a 5th generation army.

Autonomous drones can also be used for clearing run-way at airports before take-off and landing of planes; otherwise, a little piece of stone is enough to destroy the aircraft.

Technical Details of Final Deliverable

Our Quadcopter on-board sensors are a 3-axis accelerometer, both an ultrasonic and a pressure-based altimeter, but also a gyroscope and a magnetometer for measuring the quadcopter’s angular velocities and absolute orientation. Their readings are combined with the optical flow determined from a downward camera to estimate the quadcopter’s translational velocities at short intervals and with reduced drift. A second forward-facing camera provides images with a maximum resolution of 1280 × 720 pixels at a rate of up to 30 Hz for remote processing. However, in order to transfer the video stream more steadily and reduce compression artefact’s, we limit resolution and frame rate to 640 × 360 px and 15 Hz respectively. The remote board connects to the quadcopter via Radio signals. At this, our base system employs Robot Operating System, a widely used open-source middleware. A supplementary driver conveniently provides the AR. Drone’s camera images and sensor measurements, and accepts normalized control commands ? [?1, 1] for each of the quadcopter’s four degrees of freedom. Because our methods for environmental perception generate metric deviations between the quadcopter’s current and target 3D position and yaw angle, we use discrete-time PID controllers to convert them to the required normalized commands. We calculate the respective parameters, which have been determined via the closed-loop Ziegler-Nichols method: In individual experiments, proportional-only control is used to hover at a certain altitude or yaw angle, or exactly above a longitudinal or lateral line on the ground. Both values are used in a heuristic rule for finding PID control parameters which achieve quick settling without overshoot. One last part of our base system is image undistortion. it is required since all our methods for environmental perception expect the pinhole camera model to be valid. This model greatly simplifies the re-projection of image points into 3D, and approximately holds for the AR.Drone’s downward camera. Its forward-facing camera however shows significant barrel distortion, which can be described and corrected using the Brown-Conrady model. The undistorted images cover a 64? horizontal field of view on 736×360 pixels. Undistortion is our system’s only task to exploit data-parallelism through multiple CPU cores or optionally a GPU in order to minimize latency.

• Sparse 3D reconstruction may be used continuously during regular flight and therefore is our preferred method of perception. It usually yields the spatial locations of few hundreds of distinct image points, whereat their accuracy largely depends on the quadcopter’s motion.

• Dense 3D reconstruction can alternatively provide an estimated distance for most of the 265.000 pixels of an image, but in return requires exclusive flight control to virtually create a vertical stereo camera through a change in altitude

Final Deliverable of the Project

HW/SW integrated system

Core Industry

IT

Other Industries

Medical , Agriculture , Food , Transportation , Security

Core Technology

Artificial Intelligence(AI)

Other Technologies

Robotics, NeuroTech

Sustainable Development Goals

Decent Work and Economic Growth, Industry, Innovation and Infrastructure, Partnerships to achieve the Goal

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com