Hajj Crowd Monitoring and Analysis

Hajj is an annual Islamic pilgrimage to the city of Makkah, and with the ever-increasing population of the world, the number of attendees has been rising to quite some extent with an average of more than 2 million people performing this pilgrimage. With such a crowd comes extensive se

Project Title

Hajj Crowd Monitoring and Analysis

Project Area of Specialization

Computer Science

Project Summary

Hajj is an annual Islamic pilgrimage to the city of Makkah, and with the ever-increasing population of the world, the number of attendees has been rising to quite some extent with an average of more than 2 million people performing this pilgrimage. With such a crowd comes extensive security monitoring, both in person and surveillance via cameras. Our aim in this project is to automate the process of surveillance via cameras, as it has become difficult to effectively monitor crowds for potentially dangerous situations. These may include a person tripping over or a stampede that may be caused by panic in dense crowds and even predicting the likelihood of something like the aforementioned things happening. Our goal is to use machine learning to monitor these dense crowds and extract information from the live data and process it to give us information about various aspects.

Below is the schematic, which shows the layout of the Hajj locations in Makkah Mukarramah.

The Tent City of Mina , the Arafat plain and the Muzdalifah Plain lies in a triangular configuration. All three locations are connected by roads and these roads carry in excess of 2 million people from one location to another in the span of around 10-12 hours. Also notice that these roads are used by essential services providers as well. Any congestion on these roads, or crowd flow in opposite directions can lead to a potentially catastrophic situation. The entire plain is heavily monitored by video cameras, human operators look at these videos and look for potentially dangerous scenarios. Using AI and ML based techniques we would like to automatically generate alarm and preventive steps for potentially dangerous situations. We aim to detect congestion in crowd flow, flow in opposite directions and possible crowd collisions in the video data, using all the video streams available at the control center. The videos will be processed in a centralized manner and possibly points for preventive actions will be predicted along with the potential remedial steps.

Project Objectives

1. Acquire video footage of the pilgrims at various sites from the CCTV surveillance cameras.

2. Test various state-of-the-art crowd counting & crowd velocity algorithms on our hajj crowd data and short-list the best performing ones to further improve/change as per our application.

3. Use FlowNet2.0 to calculate crowd velocity in all directions by seeing the outcome when we run our data on it and tweak it to match our requirements and make it more efficient.

4. To calculate crowd count and crowd density, NWPU algorithm will be used after fine-tuning for the most efficient count. 

5. Predict, to a high degree of accuracy, the crowd density, velocity, clustered average speed & direction, bucket speed and running average on the crowd from the recorded videos of CCTV camera footage.

6. Create a model using all the outputs which calculates possible crowd collision points, the respective crowd diversion points, relative infrastructure development and generate alarms for the possible catastrophe locations in real-time running on the real time footage being received through all the security cameras.

7. Create a monitoring system that is cost effective and efficient by making it autonomous, hence the need to hire several dozen of security personnel to manually monitor the cctv footage will not be required.

8. To make it financially feasible as it will be a one time investment which will not require the purchase of any type of equipment as we plan to set up the system on their existing cameras.

9. Leaving the pilgrims unaffected and unstrained when the system is live and working

Project Implementation Method

To find the velocity of pilgrims without the addition of any hardware sensors, we use the method of Optical Flow. Optical Flow finds the motion of objects between consecutive frames of sequence.

We used FlowNet 2.0 for the implementation of Optical Flow as it was the latest, most accurate and successfully tested method to date. Since this works off of the displacement of pixels, we have to assume that the intensity remains constant between frames so that the pixel can be identified. Hence, changes in lighting such as from shadows in successive frames can cause inaccuracies. However, in our situation, the shadows from buildings, trees etc. will be negligible as these changes happen over a period of hours, and our algorithm.

The next part of the setup is to find the ratio between the number of pixels in the frame to how many meters it represents. We do this because Optical flow gives us velocity in pixels/frame, whereas we need it in real velocity, which is m/s or km/h etc. The denominator, frame, can easily be converted into seconds as we know at what Frames Per Second (FPS) the camera shoots at. Converting pixels to meters however requires this manual annotation. To achieve this, we input a line at the starting and ending of the road; the length of the line in pixels will represent M meters. The purpose of taking two lines from opposite ends are to keep into consideration the depth of the field. The end of the road may be further away from the camera and so fewer pixels will represent the same M meters as compared to the starting of the road. To find the number of pixels anywhere in the middle of the road, we use the following formula. Alpha is a point in the road between the two marked lines x & y.

For Crowd Counting, there were several open-source, state of the art algorithms was available. We tested many algorithms but chose NWPU as it gave us the least amount of error in an annotated dataset. NWPU is a Large-Scale Benchmark for Crowd Counting. This algorithm, like many computer vision based deep learning algorithms, uses CNNs to process images and find a count in each image. The dataset used for training is very diverse which can be reflected in good accuracy on the Hajj dataset which is quite unique in its build.

Our goal was to cluster people based on the direction they were moving in. We clustered the velocities based on 8 directions, top, top-right, right, bottom-right, bottom, bottom-left, left, top-left. And another cluster for stationary pilgrims to make a total of 9 clusters. We were not able to successfully use the mainstream algorithms such as K-Means or DBSCAN as it would provide incomparable results between different frames. For this we require an exact location of each pilgrim in the frame and since our crowd controlling algorithm is unable to provide us with that, we manually annotated the heads of pilgrims in a crowd to test our method.

Benefits of the Project

Our project has one major social benefit to it and that is better monitoring which implies better security and safety of the pilgrims. People will feel safer when performing the pilgrimage as an effective, fast and efficient automated system will be in place to keep a check on potentially dangerous stampede situations that can potentially cause large scale deaths. Moreover, the authorities will benefit with use of such advanced algorithms for efficient crowd surveillance and monitoring to ensure security with ease and minimum cost considering the fact that all existing resources i.e CCTV surveillance cameras are being used. By making use of these algorithms, the authorities will require much less manpower than is currently exhausted for the operations of the current monitoring system which lacks efficiency in terms of stampede detections leading to large scale loss of lives and disturbances during Hajj. Using efficient machine learning algorithms, our automated systems will predict possible stampede situations and generate alarms timely using stats indicating real-time crowd count, crowd density and crowd velocity in all directions, so authorities can take action to ensure smooth flow of crowd by timely overcoming any haphazard situations. The FlowNet 2.0 optical flow algorithm for crowd velocity and NWPU algorithms for crowd counting used in this project can be further fine-tuned and developed to implement on other similar situations where crowd needs to be monitored such as crowd at Data Darbar in Lahore and Gurdwara Darbar Sahib Kartarpur. However, some ethical issues arise in the collection and use of surveillance videos of pilgrims through CCTV cameras without their consent but nevertheless, this data is only used to test and run algorithms and will not be published for the public. The showing faces will be blurred for snippets used in any publications, presentations and documentations. However, another issue that might arise is that the further processing of the dataset requires annotation of the data; it will be used to train the algorithm to compute crowd density and flow. All of this will be done without the consent of people being used for annotation.

Technical Details of Final Deliverable

There will be an automated system which will have the ability to accurately approximate the speed of the pilgrims, the number of pilgrims in the video frame and cluster them based on the direction they are moving in. Using the aforementioned variables, it will calculate the density of people at any given place in the video frame and will predict any stampede that might occur and raise an alarm for when it does happen. The system will operate on the live feed of the existing cctv cameras at all locations on the hajj plane thereby compute live analytics for the system.

Since this is a simple crowd monitoring algorithm, upon further fine tuning, not only can we make it perform better at the hajj plane for which this project is made for, but it can work in just about any place where there are crowds. These crowds can be from different processions, rallies, a market place where people throng. Our algorithm will work in those places given there is a live feed of the cctv footage for that area and with some alterations based on the location, it can predict stampedes in those places as well.

The algorithms that we have used are FlowNet 2.0 & NWPU, both of which are in python and can run in just about any environment. We ran our code in Google Colab which is a cloud-based compiler that has every machine learning library and cloud based gpus that perform good. However, there is one technicality that must be addressed, if FlowNet 2.0 is run on colab, then the Tesla K80 GPU will not work, any other gpu will work fine. FlowNet 2.0 & NWPU are open source algorithms which mean that they are free to use for any purpose and there will be no copyright issues. Since we used google colab to run our algorithms, extra cloud storage cloud storage will be required when the system is gone live.

The end user, or the security wing of the hajj committee, will only require a one-time setup of our algorithm on all of their cameras, after which it will work on its own. Neither the pilgrims, nor the security wing will require any additional equipment for the system to work.

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Security

Other Industries

Core Technology

Artificial Intelligence(AI)

Other Technologies

Sustainable Development Goals

Decent Work and Economic Growth, Industry, Innovation and Infrastructure, Partnerships to achieve the Goal

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com