Robot path planning and optimization in virtual reality

Project Title

Robot path planning and optimization in virtual reality

Project Area of Specialization Augmented and Virtual RealityProject Summary

Robotics play prominent role in the era of the digital economy and are affecting production and employment across a broad range of industries. Pakistan as a developing country is striving hard to catch up with the pace of developing countries of the region and therefore is considering the field of robotics as one important area of R&D in different private and public sector universities and research institutions.

An important phase in the field of robotics is utilizing the robots in local industries that are autonomously controlled, re-programmable and can handle many tasks. Synthetic Products Enterprises Limited (SPEL) has become the first Pakistani industrial group to have installed robots to work its assembly lines. A few others groups are also progressing towards robotic assembly lines.

The key factors involved in industrial robots is their programming and safe interaction that require skilled persons in the field of robotics. In Pakistan, however, we lack such skilled people at present. Our project aims to use Virtual Reality to provide the support robot programming to everyone without the need of knowing programming languages or having extensive skills in robotics. Moreover, robot programming with immediate visualization of their behavior against given instructions can add great easiness and flexibility for the programmers. Robots can be enhanced with additional virtual capabilities, or can interact with humans without sharing physical space. We aim to commercialize our application to support modern needs of Industry 4.0

Project Objectives

The first primary aim of our project is to develop interactive virtual prototype using modern state of the art virtual reality methods. This interactive virtual prototype will be multisensory and employed to review, test and modify robot tasks. The project also aims to dynamically simulate complex industrial scenarios including, but not limited to, assembly, inspection, testing and maintenance situations. Moreover, the project will also be used to train industrial human-resource to better perform their respective jobs.

The second objective of our project is to provide easiness and flexibility to program variety of robotic domains that may follow different robot programming standards.

The third objective of our project is to aid in classroom and lab teaching of robotics. Teaching robotics is a challenge in many universities due to the mathematical concepts and visualization involved in it. Moreover, performing experiments in real conditions is usually expensive in terms of time, money and energy, as it requires expensive infrastructures that are generally difficult to maintain in good conditions. Using Virtual Reality platform, students can interact with industrial robot simulator and will know the effects of their own designed trajectories on several different robotic arms and cell environments without having to buy all of them and being safe of damaging the cell components. Moreover, even having a real robotic arm available for students, with this proposed VR method, all the students have the opportunity to manage and learn his own version of the robotic cell, without waiting times generated by having less robotic arms than students in classroom.

Project Implementation Method

There are three main parts of our project i.e. physical robot, VR mechanism and interface between robot and VR. The implementation scheme is described as under:

1. For robot, we will be having a 5 or 6 DOF robotic manipulator having a gripper as end effector.
2. For the VR part, we will be using Oculus Rift as VR headset to implement VR mechanism.

The Oculus Rift headset uses an OLED panel for each eye. Both panels have a resolution of 1080×1200 @ 90Hz and uses lenses that allow for a wide field of view. A Slider on the bottom of the device is used to adjust the separation of the lenses, in order to accommodate the focus of the user.

1. For the interface between robot and VR headset we will be using Unity® Game Engine platform. This will be accomplished is these steps:

1. 3D Model of the robot will be created in Solid Works® to match the exact physical dimensions of links and exact rotating capacities.
2. A virtual environment will be created in Unity comprising of a lab room and necessary lab props.
3. 3D model created in first step will be imported into Unity.
4. The links of the robot will be tested and controlled using Unity input controls.
5. Oculus Rift will be interfaced with Unity incorporating all sensors.
6. Control of the robot will be shifted to Oculus Rift using Unity Engine physics and applying inverse kinematics of the robot.
7. The calculated angles will be fed to the robot motors by interfacing Arduino Board with Unity.
8. Machine learning techniques will be used to optimize the path planned.
9. Machine vision-based mechanism for sorting of different color objects will be applied.
10. The other user-friendly features as mentioned in final deliverables will be added then.

Benefits of the Project

1. Ease to program a robot and flexibility to deal with multiple robotic platforms are the major benefits of our project. In Pakistan, industries have started using robotic manipulators for last few years. Programming of a robot for a specific task is one major concern in these applications. Industries need experts with robotic and programming specialization to get this done. Our project provides opportunity for industries to program the robotic manipulator by low-skilled person who just knows the basics of operations. Even the existing operators can very easily use this system to program the robot.
2. Updating the path plan of the robotic manipulator requires a reasonable amount of time as the system is shutdown for new path planning. New path plan is tested many times and then is transferred to actual system. Our project allows industries to reconfigure their scenarios offline while having the visualization of the real scenario without need of shutting down the actual system. The scenarios can be tested as many times as required without shutting down the system and once finalized can readily be transferred to actual system. This will improve the productivity by reducing the testing time.
3. By using Machine Learning techniques in our project, a robot is trained for complex tasks and it will self-optimize itself to maximize the productivity.  If given scenario gets modified a little, the robot will adopt to that all by itself without any extra training.
4. Our project exhibits safe human-robot collaboration which is very important for robotic applications in industry.
5. This system can be used for the training of the staff to operate the robotic system.

Technical Details of Final Deliverable

The final deliverable of our project is fully featured smart application to interact with Robot via virtual reality to complete following tasks.

1. Real-Time/Online Programming:

In this mode, the user will interact with the robot in virtual environment by gripping the end-effector via hand held controller of Oculus Rift. The user will move the end-effector to desired path and different actions like gripping/releasing the object will be controlled by other options of hand-held controller. The robot will perform those actions in real-time as being done in the virtual world. Moreover, the user will also be able to see the state of real-world robot in VR world on a screen through live feed.

1. Off-line Programming:

In this mode the user will interact with the robot same way as in the previous mode but here the real-world robot will not follow that in real-time. Instead the planned actions will be saved and user can test that by running it again as many times as required. Once finalized, the user will move that program to actual robot and robot will perform the same action. Another option will be to generate the program of the robot in robot-specific format for that action. That code file then can be loaded to the robot and it will perform that task.

1. Task optimization via Machine Learning:

To optimize the path planning of Robotic arm trained in VR (First Deliverable) to minimize cycle time and to learn a specific pattern once performed and then replicate the optimized path when asked so.

1. Machine Vision based Object Color sorting:

To sort small pickup objects based on their color. The robot will be trained in VR to pick any specific color and place that at a specific location. Then it will be able to pick that colored object placed anywhere inside its workspace and place it at the location specified in training.

Final Deliverable of the Project HW/SW integrated systemType of Industry Education , IT , Others Technologies Augmented & Virtual RealitySustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	80000
Robotic Manipulator	Equipment	1	45000	45000
Kinect	Equipment	1	15000	15000
Controllers	Equipment	1	5000	5000
Drivers & Converters	Equipment	1	5000	5000
Tool Box	Miscellaneous	1	3000	3000
Shipping	Miscellaneous	1	4000	4000
Printing & Stationery	Miscellaneous	1	3000	3000