Continuous gesture recognition for flexible human robot interaction

Recently, The concept of human-robot interaction raised many research interests. Instead of robots replace human workers in the workplace, human-robot collaboration is the direction that allows human workers and robots working together. Human-robot collaboration can release human workers from heavy

Project Title

Continuous gesture recognition for flexible human robot interaction

Project Area of Specialization

Artificial Intelligence

Project Summary

Recently, The concept of human-robot interaction raised many research interests. Instead of robots replace human workers in the workplace, human-robot collaboration is the direction that allows human workers and robots working together. Human-robot collaboration can release human workers from heavy tasks if an effective communication channel between the human workers and robots are established. Although the communication channel between human workers and robots is still limited, gesture recognition has been effectively applied as the interface between humans and robots for a long time.

We proposed a solution for this problem using continuous gesture recognition for flexible human-robot interaction. After recognizing the gesture robot mimic the action. The gesture recognition system is based on the state of the art VGG style neural network. The accuracy of this model reaches up to 91% on the dataset labeled as 20BN-JESTURE dataset. The 20BN-JESTURE dataset is large collection of densely-labeled video clips that show human performing pre-defind hand gestures in front of a laptop camera or webcam. The datset was created by a large number of crowd workers. It allows for training robust machine learning models to recognize human hand gesture.

The robot first identify the specific gesture and perform pre-defined action according to gesture. Initially we are using only five classes of gesture robot will perform actios according to these classes but in future we can increase these classes because this model is trained on large dataset contains more than hundred classes.

Project Objectives

The objective of this project is to create a more flexible gesture recognition system than the currently existing gesture recognition systems to establish better communication between humans and robots. By the hardware/software co-design, our robot will become able to understand the gestures of the operative/worker more easily and execute the corresponding function according to the gesture being recognized by the robot.
To accomplish this objective, we sub-divided it into smaller objectives which include:
? Low-cost Hardware Development
? Development of the fast and accurate algorithm

Low-cost Hardware Development
Our aim in this project is to develop a reliable hardware system with a relatively lower cost. The complete design was conceived on paper considering all the components and performance requirements. We selected Raspberry-pi as a base for our design keeping in view his low cost and portability. Raspberry pi provides good performance by consuming the power of 1.2 watts only. Then low-cost Hardware was developed from this design.

Development of fast and accurate algorithm
As we decided to use the raspberry pi. There was a need for an efficient algorithm that also provides better accuracy. In this regard, different algorithm-based approaches were developed and tested for execution time and accuracy. Convolutional Neural Network-based models were selected as they provide much higher accuracy.

Project Implementation Method

Development of Network architecture
The architecture consists of a combination of a Convolutional Neural Network (CNN) followed by a Long Short-Term Memory (LSTM) recurrent network. In our case, this network will focus on the detection of spatial patterns related to the position of the skeleton joints. we have a pre-trained CNN, we connect its output to the LSTM and we finish the training stage. Doing so, we get a more efficient and faster training stage than performing it in a single step.

The input data structure is shown in the block diagram. So, in our model, CNN takes into account to generate the features detected in the input block. Later, to integrate the features detected in the consecutive overlapping blocks we use an LSTM. The internal cycle of the LSTM allows the system to maintain information beyond the last T time steps. In the end, the controller performs work and give a signal to the robotic arm.

                                                                                                               

Low-cost Hardware Development
The low-cost hardware development directed us to use the System on chip (SoC) devices such as raspberry pi 3 model B. Raspberry Pi 3 model b offers 1 gigabyte of ram with a 1.2 GHz quad-core ARM Cortex A53 processor providing enough capability to coup with the less trained processing easily. The following is the hardware integration model of the system. 6 DOF Manipulator aluminum Robot Arm (hand) with Claw/gripper and Base.
Objectives:

Benefits of the Project

The major benefits which the project will demonstrate are as mentioned below
1. Release human workers from heavy tasks
2. This project will establish a flexible communication channel between robots and humans
3. This project can be used in factories and shopping malls for different tasks
4. This project can be used in military applications

Recently, the concept of human-robot collaboration has raised many research interests. Instead of robots replacing human workers in workplaces, human-robot collaboration allows human workers and robots to work together in a shared manufacturing environment. Human-robot collaboration can release human workers from heavy tasks with assistive robots. I proposed a system that can play a vital role in this regard because this can be used in factories for placing heavy things from one place to another or things that are too dangerous for human workers.

Although the communication channels between human workers and robots are still limited, gesture recognition has been effectively applied as the interface between humans and computers for a long time. In this project we develop an algorithm for continuous gesture recognition, we train our model on a large number of videos of continuous gesture. So no doubt that this project will provide a very flexible communication channel between humans and robots.

Our proposed system can be used in shopping malls because it can easily interact with customers and will provide shopping malls. This can also be used to repeatedly interact with people to build a rapport; since a shopping mall is a place people repeatedly visit, it provides the chance to explicitly design a robot for multiple interactions. For this, we can use RFID tags for personal identification. The robot will semi-autonomous, partially controlled by a human operator, cope with the difficulty of speech recognition in a real environment and handle unexpected situations.

Robotics can save war fighters' lives by detecting chemical, biologic, and explosive hazards. Robots neither fear nor are required to be fed. They do not feel distressed if the soldier next to him has just been shot. A survey found that saving the lives of soldiers was the greatest benefit and the greatest concern was the risk to civilians by the use of robots. Robots could make better war fighters. Our proposed system can be used in military applications because in a real environment it can easily understand the friendly gestures. The project delivered will be a proto-type robot that would have a continuous gesture recognition system and the ability to recognize specified classes of continuous gestures. The gesture recognition process would be fast and smooth. The robot will perform specific action according to human gesture
The following are the components used in the system.

Technical Details of Final Deliverable

The project delivered will be a proto-type robot that would have a continuous gesture recognition system and the ability to recognize specified classes of continuous gestures. The gesture recognition process would be fast and smooth. Robot will perform specific action according to human gesture
Following are the components used in the system

Rasberry Pi
The raspberry pi is a series of small single-board computers. It runs on raspbian which is based on the python programming language. Following are specs of raspberry pi

• Broadcom BCM2387 chipset, 1.2Ghz Quad-Core ARM Cortex (64Bit)
• 1GB RAM
• Bluetooth low energy (BLE) on board
• Micro USB socket 5V1,2.5A
• 5-pin MIPI Camera serial interface (CSI-2)
• 4 pole stereo output and composite video port
• Full size HDMI

TOF Camera
We are using D435i Depth Camera with a range of 10 meters and a depth distance of .105m. Its frame rate is 90fps and the resolution is 12890 x 720.

Other components in this project

1. Servo Motors
2. Robotic Arm
3. Power Supply

Final Deliverable of the Project

HW/SW integrated system

Core Industry

Manufacturing

Other Industries

Education

Core Technology

Robotics

Other Technologies

Artificial Intelligence(AI)

Sustainable Development Goals

Good Health and Well-Being for People, Quality Education, Industry, Innovation and Infrastructure, Peace and Justice Strong Institutions

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com