SketchCode
The project starts with gathering the images which the authors did manually, then the images were passed through labelimg software to create the bounding boxes. There were two models the images were trained on, Yolo and TensorFlow object detection. Both the models after training were converted into
2025-06-28 16:29:05 - Adil Khan
SketchCode
Project Area of Specialization Artificial IntelligenceProject SummaryThe project starts with gathering the images which the authors did manually, then the images were passed through labelimg software to create the bounding boxes. There were two models the images were trained on, Yolo and TensorFlow object detection. Both the models after training were converted into their miniature versions to be integrated into the API (flask and fast). After the integration, the models were to be deployed on AWS ec2 instances. The prediction of the models was then converted into code which is to be outputted to the user. In the end, a web app will be made in order for the client-server interaction.
Project ObjectivesTo collect data of multiple sketches along with their respective codes
• To train an application on gathered data by selecting an appropriate model to achieve maximum accuracy.
• Create an application that translates a pencil sketch of a UI into its corresponding code.
- Data and Tools Exploration A dataset has been created which contains the sketches and then we have created the corresponding codes for those sketches. Jupyter Notebook and VS Code have been used for this.
- Data Cleaning and Feature Engineering The image of the sketches has normalized and then we have augmented the images and optimized the code for the sketches so they can be referred to independently.
- Model Training An efficient model has been chosen after testing different models among Transfer learning or a custom model. We will use Tensorflow for both YOLOv4 and the custom model. At first local machines will be preferred, if the results are not good then a cloud based approach (AWS) will take place. The best accuracy has preferred as later on the codes will be automatically generated without any manual interaction.
- Comparing Results and Performance Analysis A comparison has been done on the basis of output code. The UI has been tested against the sketched UI manually.
- Deploying The proposed model has been deployed as a web service in which the client can upload a sketch and the code will be generated with the UI.
Benefits of the ProjectAn early stage of developing user-facing applications is creating a wireframe to lay out the interface. The design process begins with collaboration on a whiteboard where designers share ideas. Once a design is drawn, it is then manually translated into a working wireframe. After the wireframe’s creation, it is given to a developer to implement in code. Developing boilerplate user interface code is time-consuming work but still requires an experienced developer and hence the overall cost and time increase. Sketch2code can give faster iteration, which means that a wireframe can move to a website prototype with only the designers’ involvement. This can give accessibility to non-developers to create applications. Now anyone with little or no knowledge of coding can create great-looking wireframes. The project has also removed the requirement on a developer for initial prototypes, allowing developers to focus on the application logic rather than boilerplate GUI code. With advanced regularization techniques, we can get better results in terms of output GUI. In this way, the designers can themselves make the GUI code and hand it over to the developers for coding the backend. This project reduces one whole stage of the pipeline and the corresponding costs.
Technical Details of Final Deliverable- An efficient model has been chosen after testing different models among Transfer learning or a custom model. We will use Tensorflow for both YOLOv4 and the custom model. At first local machines will be preferred, if the results are not good then a cloud-based approach (AWS) will take place. The best accuracy has preferred as later on the codes will be automatically generated without any manual interaction.
-
A comparison has been done on the basis of output code. The UI has been tested against the sketched UI manually.
-
The proposed model has been deployed as a web service in which the client can upload a sketch and the code will be generated with the UI.
| Item Name | Type | No. of Units | Per Unit Cost (in Rs) | Total (in Rs) |
|---|---|---|---|---|
| Total in (Rs) | 75000 | |||
| AWS cloud | Equipment | 4 | 10000 | 40000 |
| Ram cost | Equipment | 4 | 7000 | 28000 |
| Marketing | Miscellaneous | 1 | 7000 | 7000 |