Efficient apache spark-based approach for a probabilistic entity resolution

Abdelkrim Tabet

doi:10.1007/s10115-025-02609-w

Return

Efficient apache spark-based approach for a probabilistic entity resolution

2025-10-15

PRE

Abdelkrim Tabet *

Abdelhamid Malki

Djamel Berrabah

DOI：10.1007/s10115-025-02609-w

Original

Original request for help

Save

Abstract

En 中文

Entity resolution (ER) is a fundamental task in data integration that aims to identify objects corresponding to the same real-world entities across multiple datasets. The blocking technique plays a crucial role in ER, as it groups dataset objects before performing comparisons. Its primary objective is to overcome the quadratic complexity of the Cartesian product, which becomes impractical as the size of the data increases. Existing blocking methods in the literature typically assign objects to blocks in a strict or binary manner, where each object is confidently placed to a block. However, this can result in objects being assigned to incorrect blocks, leading to low precision and reduced effectiveness. In this paper, we propose a novel approach to the blocking step in ER designed to overcome the limitations of binary assignment. Our method leverages the fuzzy k-modes algorithm that assigns each object a degree of belonging to each block. We adapt the fuzzy k-modes algorithm, originally oriented toward categorical data, to handle alphanumeric data in the context of entity resolution. Furthermore, we parallelize both the fuzzy k-modes algorithm and the entire blocking approach using Apache Spark, improving scalability and performance. Our experiments, conducted on both real-world and synthetic datasets, demonstrate that our method significantly improves effectiveness compared to strict or binary assignment techniques while also achieving strong efficiency on large datasets.

Keywords:

Entity resolution

Blocking

Fuzzy clustering

Probabilistic assignments

Apache Spark