Demand-Aware Erasure Coding for Distributed Storage Systems

Jun Li

doi:10.1109/TCC.2018.2885306

Return

Demand-Aware Erasure Coding for Distributed Storage Systems

2021-04-01

PRE

Jun Li *

李葆春 (Baochun Li)

DOI：10.1109/TCC.2018.2885306

Original

Original request for help

Save

Abstract

En 中文

Distributed storage systems provide cloud storage services by storing data on commodity storage servers. Conventionally, data are protected against failures of such commodity servers by replication. Erasure coding consumes less storage overhead than replication to tolerate the same number of failures and thus has been replacing replication in many distributed storage systems. However, with erasure coding, the overhead of reconstructing data from failures also increases significantly. Under the ever-changing workload where data accesses can be highly skewed, it is challenging to deploy erasure coding with appropriate values of parameters to achieve a well trade-off between storage overhead and reconstruction overhead. In this paper, we propose Zebra, a framework that encodes data by their demand into multiple tiers that deploy erasure codes with different values of parameters. Zebra automatically determines the number of such tiers and dynamically assigns erasure codes with optimal values of parameters into corresponding tiers. With Zebra, a flexible trade-off between storage overhead and reconstruction overhead is achieved with multiple tiers. When demand changes, Zebra adjusts itself with a marginal amount of network transfer. We demonstrate that Zebra can work with two representative families of erasure codes in distributed storage systems, Reed-Solomon codes and local reconstruction codes.

Keywords:

Distributed storage system

demand skewness

erasure coding

reconstruction

storage overhead

Reed-Solomon code

local reconstruction code

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.