.png){kind=icon}
Return
Exploring and cleaning big data with random sample data blocks
.png){kind=icon}
OA
AI
DOI：10.1186/s40537-019-0205-4.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Data scientists need scalable methods to explore and clean big data before applying advanced data analysis and mining algorithms. In this paper, we propose the RSP-Explore method to enable data scientists to iteratively explore big data on small computing clusters. We address three main tasks: statistical estimation, error detection, and data cleaning. The Random Sample Partition (RSP) distributed data model is used to represent the data as a set of ready-to-use random sample data blocks (called RSP blocks) of the entire data. Block-level samples of RSP blocks are selected to understand the data, identify potential types of value errors, and get samples of clean data. We provide a theoretical analysis on using RSP blocks for statistical estimation and demonstrate empirically the advantages of the RSP-Explore method. The experimental results of three real data sets show that the approximate results from RSP-Explore can rapidly converge toward the true values. Furthermore, cleaning a sample of RSP blocks is sufficient to estimate the statistical properties of the unknown clean data.
Keywords:
Big data
Exploratory data analysis
Statistical estimation
Data cleaning
Block-level sampling
Random sample partition
Distributed
Parallel and cluster computing
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.
Journal
IF:
6.4
Papers:
1.5K
Citations:
1.1W
Organization
Cited Papers
.png){kind=icon}
ShareSave
ShareSaveShareSave
View more