Real Time Three Dimensional Reconstruction using Kinect Fusion

Project Title

Real Time Three Dimensional Reconstruction using Kinect Fusion

Project Area of Specialization Augmented and Virtual RealityProject Summary

Real-time 3D Reconstruction using Kinect Fusion uses a Kinect sensor and a gyroscope to scan real-world objects and make their interactive 3D models in real-time.

With the growing real-time applications of robotics in agriculture, industries, and homes with respect to mapping the environment and interacting with it, traditional 2D images provided insufficient data to meet the requirements which is why we need 3D reconstruction. 3D reconstruction gives 3D information of the scanned environment where the focus of our project will be on scanning objects. These scanned objects can then be 3D printed, manipulated in design software such as AutoCAD and SOLIDWORKS, or used as data set for machine learning algorithms. 

Previous techniques (Multiview Stereo, Structure from Motion, and Monocular RGB) used to perform 3D Reconstruction were either inefficient or insufficient as

• They were not in real-time.
• The reconstructions were rudimentary.
• They were light-dependent.
• They could not scan smooth surfaces.

Our project aims to perform 3D reconstruction in real-time using easily available, cost-effective equipment to make it more accessible in the future. 

Project Objectives

The objective of our project is straightforward. It is to attempt to resolve the downsides of previous techniques and perform 3D Reconstruction which is

• Dense and detailed
• In real-time
• Able to detect smooth surfaces
• Able to scan object in variable lighting conditions 

Project Implementation Method

Our project will be implemented in the following steps

1. Interfacing the Kinect with software development kit (SDK)
2. Selecting a suitable programming environment and language
3. Displaying Kinect input data streams to ensure proper hardware functioning
4. Reading RGB and Depth image from the Kinect 
5. Converting RGB and Depth image to Point Clouds
6. Camera tracking using camera calibration, iterative closest point algorithm, and gyro-sensor data to fuse point clouds from all angles together into a 3D point cloud
7. Perform volumetric integration which is to join points in the point cloud into triangles
8. Perform raycasting to apply texture and make a 3D model of the scanned object

Benefits of the Project

3D reconstruction is one of the main areas of research for Computer vision. Computer Vision is a field of artificial intelligence that trains computers to interpret and understand the visual or the natural world as we humans understand and perceive it. If we can teach our robots to identify as accurately as possible the environment around them, then we can teach them how to interact with it.

Achieving the level of interaction with the environment which is similar to that of humans is a key step for the advancement of self-driving cars, to use robots as nurses in the hospitals, to compensate for the declining workforce by using robots for harvesting, to avoid working in dangerous conditions by using robots to mine minerals and to save human lives by performing accurate and precise surgeries.

Technical Details of Final Deliverable

A running environment that will reconstruct the object scanned through the Kinect in real-time. The environment will enable users to 

• Adjust depth of the scanned area
• Display RGB and Depth Image of the scene
• Display live reconstruction of the scanned object
• Create a mesh (.stl file) of the scanned object in real-time

Final Deliverable of the Project HW/SW integrated systemCore Industry OthersOther IndustriesCore Technology Augmented & Virtual RealityOther TechnologiesSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	12980
Kinect v2	Equipment	1	7000	7000
Kinect Adapter	Equipment	1	5000	5000
MPU-6050	Equipment	1	600	600
Arduino NANO	Equipment	1	380	380