Data-Aware Adaptive Compression for Stream Processing

Yu Zhang

doi:10.1109/TKDE.2024.3377710

Data-Aware Adaptive Compression for Stream Processing

2024-09-01

0

PRE

AI

DOI：10.1109/TKDE.2024.3377710

Original

Original request for help

Save

Abstract

En 中文

Stream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.3 billion, indicating a surge in data generation. However, the escalating demands for high throughput and low latency in stream processing systems have posed significant challenges due to the increasing data volume and evolving user requirements. We present a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, to significantly enhance the performance of existing stream processing solutions. CompressStreamDB utilizes nine diverse compression methods tailored for different stream data types and integrates a cost model to automatically select the most efficient compression schemes. CompressStreamDB provides high throughput with low latency in stream SQL processing by identifying and eliminating redundant data among streams. Our evaluation demonstrates that CompressStreamDB improves average performance by 3.84x and reduces average delay by 68.0% compared to the state-of-the-art stream processing solution for uncompressed streams, along with 68.7% space savings. Besides, our edge trials show an average throughput/price ratio of 9.95x and a throughput/power ratio of 7.32x compared to the cloud design.

Keywords:

Encoding

Compression algorithms

Real-time systems

Low latency communication

Data processing

Costs

Delays

Data compaction and compression

stream processing

edge computing

Data-Aware Adaptive Compression for Stream Processing

Abstract

Journal

Organization