BigData Framework for Efficient Memory Management

Project Title

BigData Framework for Efficient Memory Management

Project Area of Specialization Cloud Infrastructure,Project Summary

Implicit learning techniques like latent factor analysis has got much attention for relational learning in the last few years due to its effectiveness. However these techniques suffer with performance bottleneck due to its computation intensive nature. In order to maintain the effectiveness as well as to gain performance we aim to develop a parallel / distributed latent factor analysis technique that would appropriately map and reduce the data across cluster using Resilient Distributed Datasets of Apache Spark. We aim to compare the performance and accuracy of the proposed technique with existing technique using standard benchmark datasets. 

Project Objectives

The memory management technique to deal with big data to perform linear regression and similar predictive analysis with ease and prove to be very helpful for engineering research, business, health care, scientific research, banking & finance and machine learning where complicated statistical analysis can be performed. Analysis of large data that is very complicated for traditional analytic environment is done with ease in distributed environment without undermining on the quality of the result. Entrepreneurship these days demands the gathering of information that may extend to even petabytes. Statistics based on these customer feedback data will help expand businesses and a company that has such data to its disposal, surely has a far stronger feel on the pulse of the market.

Project Implementation Method

Would be implemented using

Scala

Apache Spark

Apache Maven / SBT

Apache HDFS

Benefits of the Project

The proposed technique would be helpful in scaling the existing latent factor analysis techniques and would accommodate big datasets.

Technical Details of Final Deliverable

Project Code  Project Manual Project Documentation

Final Deliverable of the Project Software SystemType of Industry IT Technologies Big DataSustainable Development Goals Industry, Innovation and InfrastructureRequired Resources

Item Name	Type	No. of Units	Per Unit Cost (in Rs)	Total (in Rs)
			Total in (Rs)	62000
DataBricks Server Service Based on BTU	Equipment	1	62000	62000