AR Glasses Virtual Try on

Project Title

AR Glasses Virtual Try on

Project Area of Specialization Information & Communication TechnologyProject Summary

This project is about building an AR Application that can be used in Vending Machines and shopping malls a person trying sunglasses and seeing himself/herself on screen. We will use multiple technologies with artificial intelligence models to run the application on the edge of hardware screen.

Based on AR principle, virtual glasses try-on technology achieves optimal purchase of user online glasses and quick try-on of virtual glasses, improving the senses of reality and immersion. Monocular camera is used as the input device to discuss try-on technology of AR glasses on any platform.

Project Objectives

Project objective is to solve vending machine with sunglasses customer can not touch the glasses problem with Augmented Reality.

Project Implementation Method

We are using Convolutional based neural network,

First, our task is to extract the features from the frames. In order to achieve this we are using a neural network based on Tensorflow.js .We will be extracting 468 key points from a face.

After extracting these key points this will result in formation of a mesh, A mesh is just a structure consisting of all key points. Each point is in 3D so they have three coordinates for each location. After extraction we will pass these key points to our REACT side and will be integrating our face key points to the key points of our 3D objects. ( glasses.).

Once we have created a mesh(detection of all keypoints). We know the locations of all key points we can slice and specific key points. And pass it our REACT side for stitching a 3D object to a specific location on the face.

Benefits of the Project

1. Vending machine with sunglasses. Customer can not touch the glasses but have AR experience of trying the sunglasses.
2. Can be deployed in streets and shopping malls to try some glasses virtually and order that online and recieve delivery at door step.

Technical Details of Final Deliverable

In REACT side we run ai functions and pass the video frames and we get two points with three coordinates x,y and z, then process the coordinates to derive final position, scaling and rotation of the glass in 3d and we  pass that values to Three.js to render glass model on the canvas with video frames from camera and create canvas in three scene and will be displayed to the webpage.

We have to calculate 3 different aspects given below:

• Position:
  • We take the center point of the forehead point to use as 3d glasse's position
• Rotation:
  • In rotation there are three axes, Roll,pitch and yaw.
  • Roll is calculated by determining angle between the points 3 & 4 on XY Plane ii) Pitch is calculated by determining angle between the points 3 & 4 on YZ plane
  • Yaw is calculated by determining angle between the points 1 & 2 on ZX Plane
• Scaling:
  • Scaling affects the glass size with respect to the position of the user from the camera, we use distance between points 3 & 4 to calculate the scaling of the glass as it increases and decreases while the user moves toward and away from the camera.

Final Deliverable of the Project Software SystemCore Industry ITOther Industries Finance Core Technology Augmented & Virtual RealityOther Technologies Artificial Intelligence(AI), 3D/4D PrintingSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	64000
3D models	Equipment	4	16000	64000