AEGIS VIRTUAL SELF DEFENSE TRAINER

Crime-rate and acts of physical violence in third world countries like Pakistan are alarmingly high, especially acts of violence against women. Therefore, the need to provide self-defense training arises in such societies. However, self-defense training centers are a rare occurrence in Pakistan. If

Project Title

AEGIS VIRTUAL SELF DEFENSE TRAINER

Project Area of Specialization

Computer Science

Project Summary

Crime-rate and acts of physical violence in third world countries like Pakistan are alarmingly high, especially acts of violence against women. Therefore, the need to provide self-defense training arises in such societies. However, self-defense training centers are a rare occurrence in Pakistan. If such centers are accessible, lack of quality trainers and conservative family mindsets restrict people from learning self-defense. A solution to this inaccessibility is via online courses and YouTube videos. However, such videos are non-interactive and do not provide any feedback to the performed moves or poses. In order to tackle this issue, there exist haptic suits which provide a realistic feel. However, such suits are not available locally and are way too expensive. Furthermore, these suits do not come with self-defense specific user action feedback i.e. rating the actions of the users. Therefore, the need arises to provide a system for people who don’t have access to self-defense institutes, such that it is accessible and provides feedback for improvement.

We are designing a virtual trainer. The user will be able to interact with a virtual trainer via haptic feedback and a virtual reality screen. Using low power electric pulses of different shapes and wave characteristics muscles will be stimulated (EMS: Electrical Muscle Stimulation), such that punching the air at a particular point would feel like punching a virtual character.

The user movements are being captured using a camera. A stick figure of the user is generated using the OpenPose library, which is an open source platform that runs on convolutional neural network. The key points or the coordinates of joints are extracted and compared with bench mark poses and assigned scores. The scores decide user feedback, pointing out areas that need improvement and what is done right. The poses will be also used to trigger the haptic device appropriately based on the pose. A major challenge is the varying anatomies of people. A single electrical signal characteristic cannot work for all the people. Therefore, a calibration protocol is set up, via which each user can set the desired signal parameters from trying a set of presets. The calibration is being done in signal characteristics as well as in the spatial placement of electrodes.

Project Objectives

Our project has the following objectives:

Accessibility, allowing people to easily attain the software. The hardware is aimed to be developed such that it can be attained easily as well. If not, the modularity of the hardware and software allows the software to work independently of the hardware as well. We can also have lesser hardware points for haptics (for e.g. user may opt to attain only the forced feedback mechanism).

User review should be good enough to reciprocate a human trainer’s instruction and feedback for some moves. It should also be adaptive so that the user's weaknesses and strengths are correctly identified and tackled.

And the main objective being able to provide this with minimal human interaction. The user may be able to access this product as a standalone from their homes and not need a self-defense training center and/or an instructor.

Project Implementation Method

Phase I of the project was to simulate a punch in the virtual environment (designed in Unity3D). The first step involved in this process was to estimate user's pose through a single RGB camera. OpenPose was used for pose estimation in realtime. It extracted 18 keypoints from the body using COCO model. The wrist keypoints controlled a punch mesh in the Unity world, hence the extracted keypoints had to be transformed according to Unity's world coordinates. When the software starts, OpenPose starts estimating user's pose and sends the keypoints to unity through a socket connection, and as the user moves his hand, the punch mesh in the Unity also moves. And if the punch mesh strikes a virtual wall in Unity's world, unity sends a signal to activate the hardware and the user feels the impact on his fists through EMS. The EMS device is controlled by an Arduino which is sent the commands from Unity.

Phase II of the project is more complicated; it deals with the overall software and pose evaluation. During this phase, our actual software is also being designed which also has a real-time synced database to keep track of user's profile. It has a proper authentication system too. For pose evaluation, features need to be compared between benchmark video and the user's video captured during performance. But these videos can be out of sync hence we can not directly compare the features of the respective image frames. To solve this issue, we are using Dynamic Time Warping Algorithm to align the two video sequences dynamically and temporally. This algorithm works fine only if the starting frames match. To further solve this issue, we are making use of first and second derivatives of angles (specific to movements for example, a punch-block sequence has the set of angles {'right elbow', 'left elbow', 'right shoulder', 'left shoulder'} - these angles are chosen keeping in mind the context as elbow angles are essential for calculating correctness of a block, similarly, shoulder and elbow angles are essential for punch) to determine the starting point of the video sequences. After the alignment, we use the features such as angles, velocity and acceleration (angular and linear both), back straightness, biomechanical efficiency etc to calculate a score indicating how well the user performed in comparison to the benchmark performance.

Benefits of the Project

This project aims to make self-defense training freely accessible to people (especially women). One of the major benefits of this project is that it provides self-defense training anywhere and anytime a person wants, cutting down on institutional costs and other logistics, and even in absence of a physical trainer, it provides an experience that is very close to reality through the use of haptic devices. In addition to reducing overall costs, this project is also a breakthrough in our societies that especially restrict women from attending such schools which in some way can prove beneficial to their independence in this society. Patriarchal societies like ours have normalized the false narrative that regards women as the weaker sex, due to which our women do not even feel comfortable commuting alone without the presence of a male counterpart. They also find themselves helpless in situations that can cause them harm. Self-defense training can help boost their confidence in themselves and can teach them to respond to potentially dangerous situations. This project allows people to access and train at home to prepare themselves for emergency situations.

Technical Details of Final Deliverable

User: The user completes the log-in. This information is stored and the process continues to either a fight mode or a practice mode. The output of the user are key press, selections to login, select Fight/practice session, with/without haptics and feedback to the calibration protocol. The inputs to the user will be the haptic feedback, user feedback and visual display based on which the interaction is.

    Screens+Selection Menu: The inputs to the screens is the user key inputs (ID and password), with/without haptic and practice/fight session. This block sends the decisions as input to other blocks.

    Animating Character: This block is required to animate the avatar for the video and for the session. This is set up during the prototyping and will output the avatar to the practice/fight sessions. The output of the pose detection is a stick figure which is sent to the animation block to animate certain body parts to display.

    Session: Fight and practice sessions. The animations will come accordingly since there are video animations only in the practice session and avatar real time animations in both the practice and the fight sessions.

    Calibration: Before these sessions if the user selects with haptics then the haptics need to be calibrated. The calibration scheme consists of an automatic protocol that takes in user’s feedback as input and uses them with the control unit to calibrate.

    Control Unit: Arduino or IoT platform. This takes in inputs from the communication block of the pose vector. Based on the pose vector it will send in triggers and the parameters to the haptic block. This will make decisions as to which haptic to activate (type and location). Based on distance from the opponent's avatar it will activate the haptics.

    Haptic Feedback: This block has two types of outputs, force/impulse feedback via electrodes. The output is directly exposed to the user. It makes the user feel the virtual environment.

    Pose Detection: This takes input images from the webcam and outputs a raw pose vector. These are sent to the feature extraction where features like velocity are extracted.

    Feature Extraction: The input is the raw pose vector from above and this block extracts features, mainly angles and their rates of change. This pose vector will be sent to the control unit, before finally sending it to the Pose evaluation or scoring block.

    Pose Evaluation: This block gets input from the feature extraction block. This pose vector is compared with the predefined poses. Based on this comparison the score is given. The score is then sent to adaptive teaching block and the user feedback block.

    User feedback: This block based on the numerical score gives feedback in natural language. Numerical values will not make sense. This will be sent to the display.

    Adaptive Teaching: This component suggests the next tutorial based on the current performance score. The input is the score from pose evaluation and the output is to the sessions.

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Health

Other Industries

Security

Core Technology

Others

Other Technologies

Artificial Intelligence(AI), Wearables and Implantables

Sustainable Development Goals

Good Health and Well-Being for People

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com