Gesture Controlled Humanoid Robot with vision in Virtual Reality

Humanoid robots have always been a topic of great interest in the field of robotics. The challenge they present in terms of stability, paired with the prospect of a mechanical human body has fascinated engineers since the 1970s. Now as the progress of technology has overcome most of the challenges r

Project Title

Gesture Controlled Humanoid Robot with vision in Virtual Reality

Project Area of Specialization

Robotics

Project Summary

Humanoid robots have always been a topic of great interest in the field of robotics. The challenge they present in terms of stability, paired with the prospect of a mechanical human body has fascinated engineers since the 1970s. Now as the progress of technology has overcome most of the challenges regarding Humanoid robot design, the next logical step is to explore their control and utilization paradigms. As a Humanoid robot is modeled on the humanbody, an intuitive control system would be to use human gestures for robot control. Thisproject is aimed towards developing a control scheme in which a pilot will control the Humanoid robot wirelessly through his own body gestures. Microsoft Kinect Sensor is used tocapture the motion of the pilot in 3D. This motion data is then processed on a PC and used to send movement instructions wirelessly to the robot. To provide the pilot with information aboutthe robot’s surroundings, a camera will be fitted on the robot whose feed will be streamed directly to the pilot. Once fully developed, this Humanoid robot control system has the potentialto be applied in fields of Disaster Management and Tele-Education.

Project Objectives

The project shall allow a humanoid robot to completely replicate the full body motion of ahuman controller while not necessarily being with or near the controller, allowing teleoperation of the robot in areas where human presence is not possible. The controller shall be able to interact with the surrounding environment via the robot and shall be receiving an immersive vision from where the robot is and what is around it as if the controller himself is present at the location in lieu of the said humanoid.

Project Implementation Method

The human body will be detected using the infra red capabilities of the Microsoft Kinect andusing its trained machine learning algorithms a human skeleton will be constructed whiletracking the human body joint positions in 3D. The joint poisitions will be used to determine the rotation angles of the tracked joints, while the angles that are not possible to be calculated shall be determined using inverse kinematics from the already present data. The found angles shall be transmitted to the robot using a python script whose sole responsibility is to control the robot and mantain it's stability, performing both angle calculation and robot control via C# would have resulted in loosing of data frames produced by kinect, the stability of the robot or both.The robot's vision shall be obtained via a 360 degrees camera mounted on the robot which shall be transmitting the feed back to the control center. The 360 image will then be stitchedwith immersion using Unity 3D and NVIDIA's GPU and viewed by the controller on an Oculus Rift headset.

Benefits of the Project

• Power Plants: The robot can be used in nuclear and thermal power plants, saving human lives from exposure and deadly temperatures.

• Mining Corporations The robot can work in mines better than humans which is more safe.

• Disaster Management Organizations: The robot can go to areas not accessible by humans.

• Astronauts & Space Organizations: The robot can be used in outer space for spacecraft and space station repair.

Technical Details of Final Deliverable

The resulting prototype will be able to completely replicate motions of the human, by first identifying the controller in the field of view of the Kinect sensor, which includes, but is not limited to, the motion of the hands, feet, head, torso and being able to walk, turn, etc. The robot shall be able to interact with the environment by grabbing and lifting various objects present in the zone of operation upon the will of the controller.

The controller shall also be able to view the surroundings of the robot in an immersive viewfrom the Oculus Rift VR headset.

The previously stated shall include NAO V5 as the humanoid robot and C#, Python and Unity3D (C#) as the programming languages for achieving the said objectives. The communication with the robot prototype shall be over Wifi as the capabilities of NAO V5 support such communication.

Final Deliverable of the Project

HW/SW integrated system

Type of Industry

Others , Security

Technologies

Augmented & Virtual Reality, Robotics

Sustainable Development Goals

Industry, Innovation and Infrastructure

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com