.png){kind=icon}
.png){kind=icon}
Return
Bounding substreams in distributed stream processing
.png){kind=icon}
PRE
AI
DOI：10.1016/j.is.2023.102251.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
A common problem in distributed stream processing is to split a stream into finite chunks of messages (substreams) and to determine their boundaries: stateful streaming operators should clear outdated state; time window operators should release output after all elements within the specified time range have arrived. Most state-of-the-art SPEs use punctuations to divide a stream into bounded substreams of messages. The punctuation approach is powerful but has limitations: it does not support cyclic dataflows, is poorly scalable in some cases due to intensive use of broadcasts, and becomes inefficient when the number of chunks or cluster size becomes significant. We introduce a new substream tracking technique called trAcker that overcomes the limits of punctuations. We experimentally evaluate the properties of trAcker in both synthetic and real-world environments. Experiments show that our technique is scalable, outperforms punctuations for a large number of substreams, and efficiently handles real-world cyclic dataflows.& COPY; 2023 Elsevier Ltd. All rights reserved.
Keywords:
Data streams
Punctuations
Watermarks
Substreams
Stream join
State management
Journal
IF:
3.9
Papers:
2.8K
Citations:
1.8K