Intelligent Big Data Management and Study Console

This Final Year Project proposes an Intelligent Big Data Management and Study Console (IBDMSC). The idea is to provide an integrated application that helps in migration, management, monitoring and archiving local, cloud and multi-cloud big-data repositories. This application will demonstrate the mig

Project Title

Intelligent Big Data Management and Study Console

Project Area of Specialization

Cloud Infrastructure,

Project Summary

This Final Year Project proposes an Intelligent Big Data Management and Study Console (IBDMSC). The idea is to provide an integrated application that helps in migration, management, monitoring and archiving local, cloud and multi-cloud big-data repositories. This application will demonstrate the migration process with big-data sets to migrate from one-host to another host, along with validation, logging and archiving. This application also enables time estimates and storage analysis for the migration process. The project proposed a layered model for the application, testing and deployment to local, cloud and multi-cloud platform and benchmarking the migration process for some standard big-datasets in such an environment.

Many factors should be considered before initiating a data migration process like type of data, infrastructure of the local server & third-party storage and etc. Failure during the process can cause a big loss to the organization. Therefore this application provides a solution to all these problems. It aims to provide a secure data transfer between local server to cloud and cloud to cloud.

Project Objectives

Data processing has become a new domain in computer science. Researchers are working on optimizing the data storage, minimizing the computation time and securing data from being stolen or deleted. Data processing includes different processes like data filtering, data cleaning, data manipulation and data migration etc. With the evolving technologies businesses are being transformed and generating an enormous amount of data which is an important asset for organizations. Standards and technology evolve continuously, so do business requirements and often they are the main drivers for migration. Such projects are not only costly and resource-intensive, but also require meticulous planning, right tools, and intensive testing to succeed. Migration of data is a process of moving data from one system to another i.e. from the legacy source systems into a new system, known as the target system or the application. The combination of the occurrence of data migration processes and the resources consumed in a data migration project results in significant amount of the IT budget for any company. 84% of the data migration projects fail to meet the expectations. These failures occur due to the difference of infrastructure between the organization storage and cloud, structure of data and etc.

This Project offers a complete data migration solution from cloud to cloud or cloud to Local Server and vice versa. It offers the data migration service on different clouds i.e. Microsoft Azure, Amazon AWS, Ali Baba, and Openstack (locally hosted), where user can transfer files or big data sets from one cloud to another without having the information or knowledge about the implementation of the transfer protocols. Application is designed in a layered architecture in which different layers are working concurrent to each other having different functionalities.

Project Implementation Method

Project was implemented using agile methodology. Since it was a research and product based project, research was carried out about Big Data, Clouds and technologies supporting reliable data migration following the research implementation of the project was started. Data sets like Million Song Data and etc. were collected for the testing of successful file transfer through the system with all the layers i.e. communication, migration, monitoring and logging working suitably. Other than this research on different cloud infrastructures and their APIs/ services was also carried out to have a clear image of how these clouds works and which cloud is better in different scenarios.

Project mainly focused on transfer of three types of data i.e unstructured (file), semi-structured (NoSQL) and structured (Relational Data). To easy understand the scenario the project was divided into cases. First case was the transfer of data from local machine to cloud. Second was the transfer of data from one cloud to another. Third was the transferring of data from locally hosted cloud (Openstack) to cloud. These cases were further divided into transferring a particular form of data that is either file structure, RDBMS or NoSQL. Each cloud had a different implementation of each data format since every cloud has a different service for it.

Data transfer was implemented on different clouds. Accounts of multiple clouds were first created on trial version or student accounts but due to limited access to the services and issues in receiving the credentials, accounts were switched to paid accounts to have full access to the services. Services included Object Storages, Relational Databases, and NoSQL Databases. Services were charged on the basic of instances created, which were needed every time data was to be transferred. Charges were also raised by the duration of service used.

 

 

 

 

 

 

 

Microsoft Azure:

 

Alibaba Cloud:

AWS Cloud:

Project Involves creation and deletion of multiple instances for testing purpose .If we use the above mentioned resources with minimum cost to bear, Project Estimate Budget Required will be around Rs. 90,000.

Note: We've considered our services cost as equipment cost. Project equipment item details restricting us to write actual costs.

Services

SQL Database

Cosmos DB

Blob Storage

API Access Charges

Services

Apsara DB

Table Store

Object Storage

API Access Charges

Services

Amazon RDS

Dynamo DB

AWS S3

API Access

Benefits of the Project

1. The project ensures reliable transfer of any type of data either structured, semi-structured or unstructured with monitoring, logging and validation.
2. The migration process requires experts that are costly for the organization. This project aims to reduce the cost of an organization.
3. Every cloud has a different structuring, management and overheads. This project has tackled all these issues at the application level.

Technical Details of Final Deliverable

Final Deliverables of the project includes the working web application. Web application will be capable of effectively transferring of data along with monitoring and logging of that data.

Final Deliverable of the Project

Software System

Type of Industry

IT

Technologies

Cloud Infrastructure, Big Data

Sustainable Development Goals

Industry, Innovation and Infrastructure

Required Resources

If you need this project, please contact me on contact@adikhanofficial.com