Augmented reality-based programming of robotic Manipulator

Project Title

Augmented reality-based programming of robotic Manipulator

Project Area of Specialization Augmented and Virtual RealityProject Summary

We are going through the fourth industrial revolution (i.e. industry 4.0) and everyone wants to be a part of this historical event. The world is moving fast and every industry to increase their production and not be left behind, is moving towards automation (that includes robotic manipulators, industrial robots, and cobots). We have seen a lot of advancements in technology in recent years, we have seen the rise of the metaverse, and the world was introduced to the technology of Virtual Reality (VR) and Augmented Reality (AR). Nowadays we can see these technologies in every sector, in every field with various applications that have made the handling of sophisticated machinery and technologies easier for us. The main difference between the two above-mentioned technologies is that in VR the user is completely cut off from the real world and is fully immersed in a virtual world, which has raised many concerns regarding the human safety factors. On the other hand, AR technology uses the real-world environment so the user can control their presence in the real world.
For our final year project, we have planned to integrate the two most important technologies of the future (AR and Robotics) into one. Our project ‘Augmented reality-based programming of robotic Manipulator’ aims at making the control of robotic manipulator easier for a common industrialist. The control, programming, and path planning of a robotic manipulator itself is a very difficult and hectic process that requires professional training for the individual to operate it. our project surpasses the training steps and using it, a normal person with no professional training can easily control the manipulator.
First, we will be developing our 3D model of the robotic manipulator and will import that into the Unity3D Game Engine that we will be using to control the actual robotic manipulator. Upon wearing the AR glasses the user will have an interface in front of him (which only he can see) that will be containing the robotic manipulator and some other options. For this, you do not need any professional training any worker in the industry can operate it because you cannot have the engineer with you in the industry all the time. The user will act by gripping the 3D model and moving his hand in the desired path he wants the robotic manipulator to follow. This way of programming a robot is known as Programming by Demonstration. This will be done in two modes. 
I)    Offline: The user will do the path planning and the results will be stored in our application. The user can replay the path for verification and once satisfied can upload that to the real physical robot that will follow the same trajectory and action. 
II)     Online: The robot will mimic the movement exactly at the same moment user will be guiding it through the AR experience.
This project makes not only the control and path planning easier, but also makes maintenance cheaper and easy for the companies.

Project Objectives

The primary objective of our project is to create an immersive interaction of a 5-DOF robotic manipulator and its operation through Augmented Reality. The major focus will be one available 5-DOF robot in our lab but we will have a library of other robotic manipulators as well. 

The second objective of the project is the flexibility and ease of control and programming of the robotic manipulator by anyone without the need of knowing programming languages or having extensive skills in robotics. The operator will be able to develop a specific path planning for the robot by interacting with the end-effector through Online and Offline modes. 

The third objective of our project is to develop a test bench so that new design parameters and complex scenarios can be tested before they are deployed. Customer validation will be accomplished by the actual implementation of their existing setup. 

An example setup is shown in Figure-1. 

Figure 1: Proposed Scheme of our Project 

Image Source: https://www.monash.edu/engineering/robotics/projects/augmented-reality-assisted-human-robot-collaborative-manufacturing 

Project Implementation Method

There are three main parts of our project i.e. physical robot, AR mechanism, and the interface between robot and AR. The implementation scheme is described as under:
1.    For the robot, we will be interfacing with the already built 5-DOF robotic manipulator that will be controlled by its controller, specifically made for the motors installed in the robot. This robot is built in our department Lab (HCR Lab: Human-Centered Lab under National Center for Robotics and Automation). The robot is shown in Figure-2:

Figure 2: 5-DOF Robot built by our Department 

2.    For the AR part, we will be using the Nreal Developer kit as AR glasses to implement the AR mechanism that will be used to interact with the robot.
The Developer Kit consists of a pair of Nreal Light glasses, a Nreal computing unit, and a Nreal Light controller. The glasses consist of 2 spatial computing cameras with 5MP resolution and sensors like IMU(Accelerometer /Gyroscope), ambient light sensor, and proximity sensor. It has built-in wifi and a Bluetooth module. The other specifications are provided as under.
           CPU: 8 Qualcomm® Kryo ™ 385 64-bit core
GPU: Qualcomm® Adreno ™ 630
RAM: 6GB
ROM: 64GB

The reason for selecting these glasses is the compatibility of its SDK with Unity3D. 
3.    For the interface between the robot and AR glasses we will be using the Unity3D® Game Engine platform. This will be accomplished in these steps:
i.    A 3D Model of the robot will be created in Solid Works® to match the exact physical dimensions of links and exact rotating capacities.
ii.    The 3D model created in the first step will be imported into Unity3D.
iii.    The robot model will be imposed on the physical robot by scanning its features.
iv.    The links of the robot will be tested and controlled using Unity3D input controls.
v.    Nreal glasses will be interfaced with Unity3D incorporating all sensors.
vi.    Hand gestures interaction with Nreal will be established so that the user doesn’t need to hold any controller or device in his hand to interact with the robot. This will provide a fully immersive experience to the user. One simple example of hand gesture control in AR is shown in Figure-3:
                                                          
Figure 3: The hand gesture control for AR

vii.    Control of the robot will be shifted to Nreal glasses using Unity Engine physics and applying inverse kinematics of the robot.
viii.    The calculated angles will be fed to the robot motors by interfacing the robot controller with Unity.
ix.    The path planning for a particular task will be stored and could be played multiple times in the future.

Benefits of the Project

• AR enables clear and faster communication between human and their robot co-workers which results in effective collaborations and a productive workspace. 

• Reprogramming of a robot is one of the biggest problems faced by industries because robots are programmed to perform a specific task and they will continue to perform the same job until someone programs them for something else which is a major issue and is very difficult which can only be handled by the experts and people specialized in this field. Our proposed project aims at the ease and flexibility of re-programming a robot without being dependent on an expert. It allows anyone to interact with the robot, reconfigure and reprogram it according to their desired jobs without causing any harm to the robot. From a business point of view, it is more beneficial for the companies as it cuts down their expenses and low-cost maintenance. 

• Human safety is one of the biggest concerns in the human-robot collaborative workspace and it is one of the most important in the industrial sphere. We integrate this factor into the industry with our project since there is no physical contact between the robot and the worker and the most important feature in our project is that while using the AR technology the worker will be aware of his physical environment unlike in the VR technology. 

• Our project provides an easy way to test complex dynamic scenarios before their implementation in the real system. This will help to reduce the testing time. Moreover, this system can be used for the training of the staff to operate the actual system. 

Technical Details of Final Deliverable

The final deliverable of our project is a fully-featured smart application to interact with Robots via Augmented Reality. The detail of the features is listed here:
•    Path planning of Robotic Manipulator
The application will be able to identify the target robot in the workspace. Interactable User-Interface (UI) elements will be augmented on that physical robot. The operator will interact with UI elements to move the links and the physical robot will respond to that. The application will have the ability to store the planned path and repeat it afterward. This deliverable has two modes:
1.    Real-Time/Online Programming:
In this mode, the user will interact with the robot through an AR device and the robot will respond in real-time. There will be multiple UI options for the interactions. One simple option is slider bars on each link that can be used to rotate that link as required. Then there will be simple buttons on the end effector to open or close it. There will be a button to move back the manipulator to its default home position. Moreover, there will be an indication of the workspace around the robot for safe limits. 
2.    Off-line Programming:
In this mode, the user will interact with the robot the same way as in the previous mode, but here the real-world robot will not follow that in real-time. Instead, the planned actions will be saved and the user can test that by running it again as many times as required. The user will be able to tweak the generated path to further fine-tune it. Once finalized, the user will move that program to the controller of the actual robot and the robot will perform the same action. Another option will be to generate the program of the robot in a robot-specific format for that action. That code file then can be loaded to the robot and it will perform that task.

Final Deliverable of the Project HW/SW integrated systemCore Industry OthersOther IndustriesCore Technology Augmented & Virtual RealityOther TechnologiesSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	70000
Nreal dev kit	Equipment	1	70000	70000