SketchCode

The project starts with gathering the images which the authors did manually, then the images were passed through labelimg software to create the bounding boxes. There were two models the images were trained on, Yolo and TensorFlow object detection. Both the models after training were converted into

Project Title

SketchCode

Project Area of Specialization

Artificial Intelligence

Project Summary

The project starts with gathering the images which the authors did manually, then the images were passed through labelimg software to create the bounding boxes. There were two models the images were trained on, Yolo and TensorFlow object detection. Both the models after training were converted into their miniature versions to be integrated into the API (flask and fast). After the integration, the models were to be deployed on AWS ec2 instances. The prediction of the models was then converted into code which is to be outputted to the user. In the end, a web app will be made in order for the client-server interaction.

Project Objectives

To collect data of multiple sketches along with their respective codes
• To train an application on gathered data by selecting an appropriate model to achieve maximum accuracy.
• Create an application that translates a pencil sketch of a UI into its corresponding code.

Project Implementation Method

- Data and Tools Exploration A dataset has been created which contains the sketches and then we have created the corresponding codes for those sketches. Jupyter Notebook and VS Code have been used for this.

- Data Cleaning and Feature Engineering The image of the sketches has normalized and then we have augmented the images and optimized the code for the sketches so they can be referred to independently.

- Model Training An efficient model has been chosen after testing different models among Transfer learning or a custom model. We will use Tensorflow for both YOLOv4 and the custom model. At first local machines will be preferred, if the results are not good then a cloud based approach (AWS) will take place. The best accuracy has preferred as later on the codes will be automatically generated without any manual interaction.

- Comparing Results and Performance Analysis A comparison has been done on the basis of output code. The UI has been tested against the sketched UI manually.

- Deploying The proposed model has been deployed as a web service in which the client can upload a sketch and the code will be generated with the UI.

Benefits of the Project

An early stage of developing user-facing applications is creating a wireframe to lay out the interface. The design process begins with collaboration on a whiteboard where designers share ideas. Once a design is drawn, it is then manually translated into a working wireframe. After the wireframe’s creation, it is given to a developer to implement in code. Developing boilerplate user interface code is time-consuming work but still requires an experienced developer and hence the overall cost and time increase. Sketch2code can give faster iteration, which means that a wireframe can move to a website prototype with only the designers’ involvement. This can give accessibility to non-developers to create applications. Now anyone with little or no knowledge of coding can create great-looking wireframes. The project has also removed the requirement on a developer for initial prototypes, allowing developers to focus on the application logic rather than boilerplate GUI code. With advanced regularization techniques, we can get better results in terms of output GUI. In this way, the designers can themselves make the GUI code and hand it over to the developers for coding the backend. This project reduces one whole stage of the pipeline and the corresponding costs.

Technical Details of Final Deliverable

• An efficient model has been chosen after testing different models among Transfer learning or a custom model. We will use Tensorflow for both YOLOv4 and the custom model. At first local machines will be preferred, if the results are not good then a cloud-based approach (AWS) will take place. The best accuracy has preferred as later on the codes will be automatically generated without any manual interaction.

• A comparison has been done on the basis of output code. The UI has been tested against the sketched UI manually.

• The proposed model has been deployed as a web service in which the client can upload a sketch and the code will be generated with the UI.

Final Deliverable of the Project

Software System

Core Industry

IT

Other Industries

Core Technology

Artificial Intelligence(AI)

Other Technologies

Sustainable Development Goals

Quality Education

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com