File aware distributed deduplication system in cloud environment

Amdewar Godavari

doi:10.1007/s11227-025-08058-0

File aware distributed deduplication system in cloud environment

2025-11-15

0

PRE

AI

DOI：10.1007/s11227-025-08058-0

Original

Original request for help

Save

Abstract

En 中文

In the cloud environment, number of clients are generating huge amounts of duplicate content, in the form of different types of files. Data redundancy is more prevalent among the same type of files, and it is negligible across different types of files. Applying the same deduplication technique irrespective of the file type results in wastage of resources, with insignificant duplicates elimination. If the same type of files is routed to the same data server, with deduplication, more duplicate content can be eliminated, but raises the storage imbalance problem. If files are distributed among data servers, ignoring the file type, better storage balance can be achieved, but the eliminated duplicate content decreases. In order to address these challenges, a distributed deduplication system is proposed. The system categorizes files into three groups—high, low, and unpredictable duplicate files—based on their redundancy levels. A group of data servers is allocated for each category of files. Preprocessing at source and resource-intensive duplicate content identification and elimination at the data servers reduces communication overhead. Data servers apply similarity-based segment-level deduplication for high and unpredictable duplicate files, and file-level deduplication is applied for low duplicate files. This system gives a high degree of load balance while achieving comparable space saving.

Keywords:

Distributed deduplication

Disk bottleneck

Load balancing

Cloud computing

Cloud storage

File aware distributed deduplication system in cloud environment

Abstract

Journal

Organization

Cited Papers