Search Engine for item comparison using web scraping

Project Title

Search Engine for item comparison using web scraping

Project Area of Specialization Software EngineeringProject Summary

E-business is the most modern form of business organization which implies intensive use of Internet technology,  while e-commerce represents the most popular part of e-business.

Lower prices - it’s well known that prices in some online stores are more economical than fares in a classic, brick-and-mortar store. Why are online prices cheaper than standard prices? Well, it’s pretty easy to explain. Merchants with an online store don’t need a physical storefront and more than two or three people who will take care of the eCommerce website. According to that, online merchants have lower costs than classic merchants and that is the main reason why they can afford lower prices of goods and services for their buyers with low shipping rates for delivery.

Faster and easier buying -  Just in a few clicks you can order desired products or services and get it on your home address

No unfriendly sellers -the reality is that sometimes sellers are not happy with all of your questions about sizes, colors, and other things you want to know about the products. Their answers sometimes could be rude- and that’s something you don’t want to hear.

Availability 24/7 -

Web scraping, web harvesting, or web data extraction: is data scraping used for extracting data from websites. The web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser.

Web scraping for price comparison is an essential tool of any business today. In the past, a usual buyer had to go through many different websites to compare prices from various e-commerce sites and get the best deals available.However, this is no longer the case in today’s world since most eCommerce businesses already have tools to monitor price changes in real-time. Retailers seize every opportunity to stay relevant in the competitive atmosphere and win more customers for their business. One of the trusted methods is using web scraping for price comparison. This process is beneficial for both a business owner and a customer. With the help of web scraping, you can create better coverage over significant e-commerce sites, and a display of greater variety for the same product, which results in a better customer experience.

Customers want the best price matching to happen, which is possible when the data is available in real-time. This happens via real-time crawling and scraping, aka live crawlers.

Live scraping is taking over other markets, such as news aggregators and digital marketing, with faster scraping techniques in programming languages such as R and Python.

Web scraping price monitoring software for eCommerce sites to compare prices with Python is generally cheaper than the major eCommerce platforms. It attracts bargain hunters, while the other caters typically to the less price sensitive.Building a dataset with Python script using Selenium and the Chrome web driver to automate the scraping process for product names and price lets you conduct data analysis.

Project Objectives

The objective of this project is to develop a price comparison website that will have the following functions:

1. To provide customers with a list of price comparison and highlight the cheapest price specifically in home groceries products.

2. To increase price consciousness among consumers.

3. To ensure that the price database is updated regularly so that customers will be able to get accurate results.

4. To provide service for users to find the product’s price.

Instead of taking hours and energy to go to each shops just want to check on the price, our website offered better solution by getting all the price and users just need to go online and choose which product they want to know and the list of retailers and the price offered will be shown. Users can check it from anywhere, no matter at home or at work, or even in the train while going back from work, our website is accessible anytime as long as there is internet connection.

Project Implementation Method

Firstly we have selected the programming language for the backend of the project. 

PYTHON Programming language is used to create the backend of our final year project which is the comparison website with the help of scrapping. We use the PYTHON programming language for our project backend because it is the most popular programming language and it can handle almost all processes related to data extraction. Python is a perfect fit for building web scrapers and extracting data as it has a large selection of libraries, and an active community to search for help if You have issues with coding. One of the most important parts why use Python for web scraping is that Python is easy to learn, clear to read, and Simple to write in.

The process to scrap data:

Firstly, we scrap data from the website with the help of a scrapper by the method explained below:

•  Firstly, we give the link then the scrapper load the document or webpage by the link. This process is called document load.
• Secondly, The Parsing process is done when the crawler finds out tags that are written in the script.
• Thirdly, The scrapper extract data from the tags of a website. This process is called Extraction.
• Fourthly, the Transformation process is done in which differently structured data sets are reconstructed, making two or more data sets compatible for further analysis.

At last, the data is stored in the database. We use the MongoDB database for this purpose.

GUI(Graphical User Interface)

We created a simple user interface with the help of the following  PYTHON Libraries:

• Tkinter
• Os

           The user interface includes a button. When the button is pressed then the Scrapper scrap the data from the websites then data is stored in the MongoDB database.

The frontEnd of the project:

we have selected the MERN package to create the frontend of the project. Mern includes

• MongoDB database
• Express
• React js
• Node js

MongoDB database:

MongoDB is a source-available cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with optional schemas. MongoDB is developed by MongoDB Inc. and licensed under the Server Side Public License.

Express:

Express, is a backend web application framework for Node.js. It is designed for building web applications and APIs. It has been called the de facto standard server framework for Node.js.

React js:

React is a free and open source frontend JavaScript library for building user interfaces based on UI components. It is maintained by Meta and a community of individual developers and companies.

Node js:

Node.js is an open-source, backend JavaScript runtime environment that runs on the V8 engine and executes JavaScript code outside a web browser.

The website contains a search bar and all the data that is extracted from the websites.  

Server:

we create a server that is used to connect the frontend to the backend with the help of the framework of node js, Express.

then we buy a domain and hosting and upload our project.

Benefits of the Project

Comparison shopping engines are channels for collecting product information, including pricing, from participating retailers and then displaying that collective information on a single results page in response to a shopper’s search query. This lets shoppers compare pricing, shipping options, and service from multiple retailers on a single page and choose the merchant that offers the best overall value.

A price comparison website acts as a platform or medium between the consumers and the sellers. It allowsconsumers to see different lists of prices for the product chosen by the user and it helps consumers to make an informed decision about which to choose to save money.  It allows consumers to see different lists of prices for the product chosen by user and it help to make an informed decision. 

Comparison sites save you money

The main reason people use price comparison websites is that they allow you to find the best deal by checking hundreds of Products from different sites. There may be a smaller or lesser-known company that offers the service you wish to buy at a much lower price. If you are comparing services like broadband plans, you can easily compare different features side by side, allowing you to save money paying for what products you don’t need. This can, of course, save you lots of money when you purchase an item. 

We also negotiate exclusive deals and offers for our customers, so you can save even more!
Comparison sites save you time

Finding the best-priced option can be very time-consuming. Visiting each e-commerce website, finding different prices checking ratings and reviews on different sites, and trying to find the right information to compare with like can be incredibly time-consuming.  Price comparison websites gather all the information together in one place – so you can easily compare the options in one place. You can also view hundreds of customer reviews and ratings which can help you make better shopping decisions.

Comparison sites offer you choices

Another great benefit of price comparison websites is that you can access hundreds of different sites in one place. In the changing global era, there may be a newer or lesser-known e-commerce store offering a great deal or service. Comparison sites have larger and smaller e-commerce sites together in one place, allowing you to make an informed decision. 

Users' ratings and reviews

Each supplier will claim that the product they are selling is the best out there. So, how can a customer distinguish suppliers with an attractive or solid reputation? User ratings and reviews. Most CSEs allow buyers to share their experiences. Nothing beats reading or hearing the advantages and disadvantages of an item from persons who have used it.

Compare details

Comparison sites normally drill down into each product’s details and even include the extras like the refund policies, delivery speeds, shipping cost, and payment security. This allows you, the buyer, to make an informed choice

Technical Details of Final Deliverable

Hardware:

• Powerfull Server Computer

Technologies:

• Python

• JavaScript

• MongoDB

Editor:

• Pycharm

• Visual Studio Code

Libraries:

Python Libraries:

• Requests (HTTP for Humans) Library for Web Scraping.

• lxml Library for Web Scraping.

• Beautiful Soup Library for Web Scraping.

• Selenium Library for Web Scraping.

• Scrapy.

JavaScript Libraries and framework:

• React

• MongoDB

• ExpressJs

• NodeJs

Components of Final Deliverable:

• Front-End: Design with MERN Stack

A website on which Products from different e-commerce sites are displayed With Prices, images, Rating, and other essential   information and a button to go to the product page

• Backend: Design With Python

With the help of python different libraries a tool (Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. The web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser.)is developed which can scrap the products from the e-commerce site and send them into the database.

• Database

MongoDB is used to store and retrieve data

• Server

3  servers from google cloud computing to target different products in different areas.

• Network Connection

A good internet connection is needed

Final Deliverable of the Project Software SystemCore Industry ITOther IndustriesCore Technology OthersOther TechnologiesSustainable Development Goals Responsible Consumption and ProductionRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	66042
domain + hosting(2-year) for front-end web application	Equipment	1	26529	26529
Google cloud platform(laaS)-3x Windows servers for backend(python)	Equipment	3	9838	29513
Flex designing & printing	Miscellaneous	5	1000	5000
Documentation	Miscellaneous	400	5	2000
Brochures & marketing stuff	Miscellaneous	300	10	3000