Three Dimensional Augmented Reality capture and display system
Augmented Reality (AR) is one of the emerging techniques due to the nature of three-dimensional vision being more involving and interactive for any individual. The AR systems are demonstrated for variety of use scenarios and a considerable work is done in this field to provide the AR capture and dis
2025-06-28 16:36:22 - Adil Khan
Three Dimensional Augmented Reality capture and display system
Project Area of Specialization Augmented and Virtual RealityProject SummaryAugmented Reality (AR) is one of the emerging techniques due to the nature of three-dimensional vision being more involving and interactive for any individual. The AR systems are demonstrated for variety of use scenarios and a considerable work is done in this field to provide the AR capture and display systems of high optical quality. However, the indigenous development of AR systems is very limited in Pakistan due to higher costs associated with the optics of such systems.
In this project, we propose an integrated 3D augmented reality system with the features of both 3D image capture and Multiview 3D display. The proposed system allows creation of environment of an observer/user that can help to view and analyze any real object/person standing at the remote place. The display sub-system provides three different views of the scene/object including front stereo-3D view and two side-views. A semi-reflective mirror based optical setup is developed to provide the notion of see-through display to demonstrate AR. On the other hand, the capture subsystem consists of four-camera capturing the front stereo-views and two side-views. The capture and display modules placed at two different physical locations are further connected through internet.
Project Objectives- Design and develop a stereoscopic and multi-view image capture setup.
- Design the optical setup of the system using geometric optics.
- Develop the AR display system using the designed optical setup.
- Develop a content capture and processing pipeline using python, OpenCV and OpenGL.
- Develop an internet-based link between display and capture modules of the system.
- Demonstrate the real-time capture and display of real objects using the developed prototype system.
The project implementation is divided into two parts;
- Hardware setup for 3D image capture and 3D AR display
- Customized software application using python programming to capture, process, store, transmit and render the images on the display.
1- 3D Capture and Display Hardware
The capture and display hardware will consist of two setups physically located at different places. The 3D image capture setup will consist of 4 cameras capturing the object simultaneously. One pair of cameras will be used to capture the front view of the object in the stereoscopic format while two remaining cameras will be utilized to capture the side-views (i.e. left and right).
images/Three Dimensional Augmented Reality capture and display system _1639949983.jpg
The 3D AR display setup will be based on a tabletop optical setup combined with liquid crystal display panels to provide 4 views. The optical setup will allow the display of capture content along with the real-world view to provide the true AR experience. The stereo 3D views will be separated by using polarization multiplexing of light. The left and right eye views will be separated by using different light polarization orientation for each eye. The side views will be displayed in the 2D AR format and will be reflected towards the observer/user using semi-transparent mirrors. The voice recording and playback feature will be also be interfaced.
images/Three Dimensional Augmented Reality capture and display system _1639949985.jpg
images/Three Dimensional Augmented Reality capture and display system _1639949986.jpg
https://ignite.org.pk/ngiri/assets//upload_ckeditor_files/2021-01-19_03:28:21pm_INTEGRATED DISPLAY SYSTEM VIEW 1.png
images/Three Dimensional Augmented Reality capture and display system _1639949987.png
(THE 3D MODELS OF THE CAPTURE AND DISPLAY ASSEMBLIES ARE GIVEN IN THE LINKS BELOW. THEY CAN BE OPENED BY 3D VIEWER.)
https://muet14-my.sharepoint.com/:u:/g/personal/17es24_students_muet_edu_pk1/EVRebcUa0e1DjDZKEZ84AdoBnb6s2AefpzTBROsuYnkWpQ?e=L6Wjn8
https://muet14-my.sharepoint.com/:u:/g/personal/17es24_students_muet_edu_pk1/EQWe0Qy8LcJHjKEzVGyScXAB-_PiQ4Zs08j-Ofb3-xu24g?e=oZaKtU
2- Software Application for Processing
The software part will consist of a customized software application to be developed in Python using OpenCV, OpenGL and related libraries. The software application will consist of two programs running on the capture setup and display setup respectively. The application programs will be responsible for capture, processing and transmission of captured views at one side and reception and rendering of views on the display at receiving side. In addition to captured real content, the 3D animated objects will also be developed using open-source Blender software.
Benefits of the ProjectThe target benefit of the project is to provide enhanced 3D capture and visualization of real objects/persons to develop the augmented reality systems at the local level using cost-effective off-the-shelf optical and electronic components as an alternative to expensive AR products.
Another main benefit of the project is to get a more interactive learning and teaching experience especially in the circumstances of COVID-19 in which attending class from home is difficult. The developed system can provide 3D perception of telepresence to provide a virtual classroom experience. The other benefits of the project include: medical visualization, online sports training sessions in which observation of angles of the body will be easily observed through display system, online Gym training sessions where body angles and position are very important, 3D augmented reality gaming experience, 3D movies, advertisement
Technical Details of Final DeliverableThe final demonstration will be based on integrated software and hardware.
The 3D image capturing setup and 3D AR display setup will be demonstrated separately. Both will be connected through the internet with the help of two PCs providing a real-time display. The software application will be implemented on both sides and demonstration will be in the form of video processing and animations along with the 3D content.
Final Deliverable of the Project HW/SW integrated systemCore Industry EducationOther Industries IT , Medical , Media , Telecommunication Core Technology Augmented & Virtual RealityOther Technologies Artificial Intelligence(AI), Internet of Things (IoT), OthersSustainable Development Goals Quality Education, Decent Work and Economic Growth, Industry, Innovation and InfrastructureRequired Resources| Item Name | Type | No. of Units | Per Unit Cost (in Rs) | Total (in Rs) |
|---|---|---|---|---|
| Total in (Rs) | 77250 | |||
| USB camera with microphone | Equipment | 4 | 3000 | 12000 |
| Liquid crystal display panels | Equipment | 4 | 4500 | 18000 |
| Prototype housing assembly for capture module | Equipment | 1 | 5000 | 5000 |
| Prototype housing assembly for AR display module | Equipment | 1 | 10000 | 10000 |
| Optical grade semi-transparent glass | Equipment | 4 | 3500 | 14000 |
| Linear polarizer sheet | Equipment | 1 | 1500 | 1500 |
| Power supply cable | Equipment | 4 | 100 | 400 |
| USB extension | Equipment | 2 | 250 | 500 |
| USB Hub | Equipment | 2 | 300 | 600 |
| USB type virtual graphics card | Equipment | 3 | 900 | 2700 |
| VGA to HDMI connector | Equipment | 1 | 300 | 300 |
| HDMI cable | Equipment | 1 | 250 | 250 |
| Tripods | Equipment | 3 | 1000 | 3000 |
| Wiring, soldering, stationary, documentation, transport | Miscellaneous | 1 | 9000 | 9000 |