.png){kind=icon}
返回
An efficient spark-based adaptive windowing for entity matching
.png){kind=icon}
PRE
AI
DOI：10.1016/j.jss.2017.03.003.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Entity Matching (EM), i.e., the task of identifying records that refer to the same entity, is a fundamental problem in every information integration and data cleansing system, e.g., to find similar product descriptions in databases. The EM task is known to be challenging when the datasets involved in the matching process have a high volume due to its pair-wise nature. For this reason, studies about challenges and possible solutions of how EM can benefit from modern parallel computing programming models, such as Apache Spark (Spark), have become an important demand nowadays (Christen, 2012a; Kolb et al., 2012b). The effectiveness and scalability of Spark-based implementations for EM depend on how well the workload distribution is balanced among all workers. In this article, we investigate how Spark can be used to perform efficiently (load balanced) parallel EM using a variation of the Sorted Neighborhood Method (SNM) that uses a varying (adaptive) window size. We propose Spark Duplicate Count Strategy (S-DCS++), a Spark-based approach for adaptive SNM, aiming to increase even more the performance of this method. The evaluation results, based on real-world datasets and cluster infrastructure, show that our approach increases the performance of parallel DCS++ regarding the EM execution time. (C) 2017 Elsevier Inc. All rights reserved.
Keyword:
Adaptive windowing
Entity matching
Load balancing
Spark
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
4.1
论文数:
5.5K
被引数:
8.4K
机构
引用论文
An Approach to Source-Code Plagiarism Detection and Investigation Using Latent Semantic Analysis
PREAI
分享收藏分享收藏A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication可扩展记录链接和重复数据删除的索引技术综述
OAAI
OAAI分享收藏Structural Studies of Thiamin Monophosphate Kinase in Complex with Substrates and Products,硫胺素单磷酸激酶与底物和产物的复合物的结构研究 ,.png)
BiochemistryOAAI
BiochemistryIF0
OAAI分享收藏Glucocorticoid Receptor Expression Correlates with Clinical Outcome in Myeloma Patients Treated with Glucocorticoid-Containing Regimens.Blood
BloodIF0
PREAI
分享收藏 没有更多内容
没有更多内容