.png){kind=icon}
返回
Exploring and cleaning big data with random sample data blocks
.png){kind=icon}
OA
AI
DOI：10.1186/s40537-019-0205-4.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Data scientists need scalable methods to explore and clean big data before applying advanced data analysis and mining algorithms. In this paper, we propose the RSP-Explore method to enable data scientists to iteratively explore big data on small computing clusters. We address three main tasks: statistical estimation, error detection, and data cleaning. The Random Sample Partition (RSP) distributed data model is used to represent the data as a set of ready-to-use random sample data blocks (called RSP blocks) of the entire data. Block-level samples of RSP blocks are selected to understand the data, identify potential types of value errors, and get samples of clean data. We provide a theoretical analysis on using RSP blocks for statistical estimation and demonstrate empirically the advantages of the RSP-Explore method. The experimental results of three real data sets show that the approximate results from RSP-Explore can rapidly converge toward the true values. Furthermore, cleaning a sample of RSP blocks is sufficient to estimate the statistical properties of the unknown clean data.
Keyword:
Big data
Exploratory data analysis
Statistical estimation
Data cleaning
Block-level sampling
Random sample partition
Distributed
Parallel and cluster computing
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6.4
论文数:
1.5K
被引数:
1.1W
机构
引用论文
An Asymptotic Ensemble Learning Framework for Big Data Analysis一种用于大数据分析的渐近集成学习框架.png)
IEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAI分享收藏
查看更多内容