Mining Big Data with Random Forests

Alessandro Lulli

doi:10.1007/s12559-018-9615-4

Return

Mining Big Data with Random Forests

2019-01-03

PRE

Alessandro Lulli

Luca Oneto *

Davide Anguita

DOI：10.1007/s12559-018-9615-4

Original

Original request for help

Save

Abstract

En 中文

In the current big data era, naive implementations of well-known learning algorithms cannot efficiently and effectively deal with large datasets. Random forests (RFs) are a popular ensemble-based method for classification. RFs have been shown to be effective in many different real-world classification problems and are commonly considered one of the best learning algorithms in this context. In this paper, we develop an RF implementation called ReForeSt, which, unlike the currently available solutions, can distribute data on available machines in two different ways to optimize the computational and memory requirements of RF with arbitrarily large datasets ranging from millions of samples to millions of features. A recently proposed improved RF formulation called random rotation ensembles can be used in conjunction with model selection to automatically tune the RF hyperparameters. We perform an extensive experimental evaluation on a wide range of large datasets and several environments with different numbers of machines and numbers of cores per machine. Results demonstrate that ReForeSt, in comparison to other state-of-the-art alternatives such as MLlib, is less computationally intensive, more memory efficient, and more effective.

Keywords:

Random forest

Random rotation ensembles

Model selection

Big data

Apache Spark

Memory efficiency

Computational efficiency

Open source software

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.