Real Time Panorama Image Stitching for Implementation in Helmet Mounted Displays of Fighter Aircraft

The F-35 Lightning-II uses a six-camera array on the Aircraft to give the pilot a view of everything around him. The pilot can see through the bottom of the fuselage as well. This system can be achieved using real time image stitching. The aim of the project is to use / develop image stitching algor

Project Title

Real Time Panorama Image Stitching for Implementation in Helmet Mounted Displays of Fighter Aircraft

Project Area of Specialization

Electrical/Electronic Engineering

Project Summary

The F-35 Lightning-II uses a six-camera array on the Aircraft to give the pilot a view of everything around him. The pilot can see through the bottom of the fuselage as well. This system can be achieved using real time image stitching. The aim of the project is to use / develop image stitching algorithm to generate panorama image view to view everything around the cockpit using helmet mounted display of pilot.

In the modern aerial warfare scenarios, dangercould come in any shape and at any time. Most of the modern Avionics developments have targeted better situational awareness for the pilot including better Radar Systems as well as IRST Systems. One of these developments is the Electro- Optical Distributed Aperture System (DAS) which uses cameras fitted around the frame of the aircraft to give uninterrupted 360 view to the pilot. The System also tracks for any flying objects in the vicinity using the IR cameras, providing IRST all around the aircraft. This gives the pilot an edge over the adversary as no one can then surprise the pilot; him being aware of the other at all times. A proof of concept personal DAS can be created using the process of Image Stitching and accelerating it to be performed in real-time.

Project Objectives

• Understanding of Image Stiching Techniques along with computer graphic processing.
• Development of Image Stitching Software for 6 cameras view (horizontal and vertical views)
• Development of Real Time Panorama Viewing Software
• Improvement of Hardware for Real Time Video View
• Implementation of Developed Software on Hardware

Project Implementation Method

A. Project Approach
The project would be divided into three phases listed below.
• Database Development for Synchronous Images • Image
Stitching Software Development • Real Time Implementation
of Panoramic viewer using optimised parameters details of the phases are given below.

•  Phase 1 : Database Development for Synchronous Images Images are to be taken simultaneously using thesix cameras. This image database would be used to find theoptimum stitching parameters.
•  Phase 2 : Image Stitching Software Development:

This software would take images from the developed data-set and stitch them together. This would provide us with stitching data that would be used in the third phase. In this software we perform the calibration, image registration and blending steps. The blending of the image can also be edited manually in this software so that we can get the best stitching parameters out of this step.

•  Phase 3 : Real Time Implementation for Panoramic

View using Optimized Parameters: This phase would take the previously calculated optimtized stitching parameters and apply them to the incoming video streams from the cameras to create a seamless picture which is then forwarded to the HMD. Seperating the parameter calculation and viewing phases helps in efficiently managing the resources as no computation would have to be performed in the real time viewing software which would hamper performance.

Benefits of the Project

Distributed Aperture System or DAS which this project is trying to imitate is a system used in Fifth Generation Aircraft such as the Lockheed Martin F-35 and the Chengdu Shanghai J-20.

Such a system should allow unparalleled situational awareness as well as uninterrupted view for the pilot allowing him to see at night as well as detect objects that he/she would normally be unable to detect at day time. The System will the concept of Image Stitching and apply hardware acceleration on it to accelerate it to run at real time. The previous implementations show that such a system is possible to deploy in next and current generation aircraft of the Pakistan Air Force.

This project presents just a method of implementing this optically but if implemented on Aircraft and using Infrared Cameras instead of Optical Cameras, this system would allow for a large variety of options. Some of them are listed below.

• Night Vision

This System would allow the pilots to see everywhere around the aircraft in the night which would eliminate the need for night vision goggles and would help in night missions and sorties.

• IRST

Infrared Search and Track (IRST) can also be done using this system. This IRST compared to traditional implementations provides tracking capabilities all around the aircraft and would inform the aircraft as to nearby aircraft as well as incoming passive missiles whose warnings may or may not be recieved on the Radar Warning Reciever.

•  Aircraft Orientation

This software can also inform the pilot of the aircraft’s orientation visually and draw a horizon line based on the incoming imagery directly on the HMD of the Aircraft. This would prevent the pilot from getting disorientated during
manuever.

• Exclusive System

The system can to be made to run exclusively on Aircraft systems with proper Helmet Mounted dependent Displays serving as the output. 

• Night Vision

IR cameras may be used instead of the optical cameras to ensure night vision as well as enhanced vision.

• Detection algorithms

Detection algorithms may also be applied on the upcoming feeds to ensure detection of enemy and allied aircraft well before time to ensure proper usage of DAS. These improvements would entail heavy load on the machinery used. Proper cooling and proper hardware is certainly a must if these improvements are to be made.

Technical Details of Final Deliverable

The project, when incorporated into a full system, would allow unparalleled situational awareness as well as uninterrupted view for the pilot allowing him to see at night as well as detect objects that he/she would normally be unable to detect at day time. The System will the concept of Image Stitching and apply hardware acceleration on it to accelerate it to run at real time. The previous implementations show that such a system is possible to deploy in next and current generation aircraft. 

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Others

Other Industries

IT , Manufacturing , Transportation , Security

Core Technology

Wearables and Implantables

Other Technologies

Sustainable Development Goals

Industry, Innovation and Infrastructure

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com