.png){kind=icon}
返回
One-Pass Inconsistency Detection Algorithms for Big Data
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2019.2898707.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Data in the real world is often dirty. Inconsistency is an important kind of dirty data; before repairing inconsistency, we need to detect them first. The time complexities of the current inconsistency detection algorithms are super-linear to the size of data and not suitable for the big data. For the inconsistency detection of big data, we develop an algorithm that detects inconsistency within the one-pass scan of the data according to both the functional dependency (FD) and the conditional functional dependency (CFD) in our previous work. In this paper, we propose inconsistency detection algorithms in terms of FD, CFD, and Denial Constraint (DC). DCs are more expressive than FDs and CFDs. Developing the algorithm to detect the violation of DCs increases the applicability of our inconsistency detection algorithms. We compare the performance of our algorithm with the performance of implementing SQL queries in MySQL and BigQuery. The experimental results indicate the high efficiency of our algorithms.
Keyword:
Inconsistency detection
big data
one-pass algorithm
data quality
denial constraint
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
.png){kind=icon}
查看更多内容